Sep 08 11:28:12.565326 ip-10-0-139-41 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 11:28:12.565331 ip-10-0-139-41 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 11:28:12.565338 ip-10-0-139-41 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 11:28:12.565554 ip-10-0-139-41 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 11:28:22.148001 ip-10-0-139-41 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 08 11:28:22.148017 ip-10-0-139-41 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 11305d7c476c4372a3824b2a6e277e36 --
Sep 08 11:29:53.328765 ip-10-0-139-41 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 11:29:53.750140 ip-10-0-139-41 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 11:29:53.750140 ip-10-0-139-41 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 11:29:53.750140 ip-10-0-139-41 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 11:29:53.750140 ip-10-0-139-41 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 11:29:53.750140 ip-10-0-139-41 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 11:29:53.750836 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.750195    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 11:29:53.757004 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.756988    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 11:29:53.757004 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757003    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757007    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757012    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757017    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757021    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757025    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757029    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757032    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757035    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757038    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757041    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757044    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757052    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757055    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757058    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757061    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757065    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757068    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757071    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 11:29:53.757152 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757075    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757078    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757081    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757084    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757087    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757090    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757093    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757095    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757098    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757101    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757103    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757106    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757108    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757111    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757114    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757117    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757120    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757123    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757126    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757130    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 11:29:53.757628 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757133    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757135    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757138    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757141    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757143    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757146    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757149    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757151    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757154    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757157    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757159    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757162    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757165    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757168    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757170    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757173    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757177    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757179    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757182    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757185    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 11:29:53.758105 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757187    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757190    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757193    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757195    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757198    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757200    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757204    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757207    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757209    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757212    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757215    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757218    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757220    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757223    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757227    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757230    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757233    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757236    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757238    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 11:29:53.758616 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757242    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757244    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757247    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757249    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757252    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757255    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757257    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757260    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757262    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757265    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757269    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757271    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757274    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757278    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757280    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757283    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757286    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757288    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757291    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.757293    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 11:29:53.759069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758927    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758937    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758940    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758943    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758946    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758950    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758953    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758956    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758958    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758961    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758964    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758967    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758969    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758972    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758975    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758977    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758981    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758985    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758988    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758991    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 11:29:53.759581 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758994    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.758997    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759000    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759003    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759005    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759008    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759011    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759013    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759016    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759018    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759021    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759024    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759026    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759029    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759031    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759034    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759036    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759039    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759047    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759050    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 11:29:53.760106 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759053    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759055    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759058    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759060    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759063    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759065    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759068    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759071    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759074    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759076    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759079    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759082    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759084    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759087    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759091    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759093    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759096    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759099    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759101    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759104    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 11:29:53.760605 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759106    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759109    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759112    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759114    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759117    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759119    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759122    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759126    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759128    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759131    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759133    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759141    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759144    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759146    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759149    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759151    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759154    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759157    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759159    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759162    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 11:29:53.761089 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759164    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759166    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759169    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759172    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759175    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759177    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759180    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759183    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759185    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759188    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759193    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759196    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759200    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759203    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759207    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759210    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759212    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759215    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759218    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 11:29:53.761593 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759666    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759671    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759675    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759678    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759687    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759690    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759693    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759696    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759698    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759701    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759704    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759706    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759709    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759711    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759714    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759716    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759719    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759723    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759728    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759731    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 11:29:53.762053 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759734    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759737    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759740    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759743    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759746    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759749    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759752    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759755    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759758    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759761    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759764    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759766    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759769    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759772    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759774    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759778    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759780    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759788    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759792    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 11:29:53.762542 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759794    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759797    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759799    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759801    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759804    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759806    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759809    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759812    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759815    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759817    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759821    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759825    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759828    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759831    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759833    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759836    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759838    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759841    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759844    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 11:29:53.762999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759846    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759849    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759851    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759854    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759856    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759859    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759861    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759864    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759867    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759869    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759872    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759874    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759881    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759884    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759887    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759889    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759892    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759894    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759897    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759899    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 11:29:53.763462 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759904    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759906    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759909    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759911    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759914    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759917    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759919    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759922    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759926    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759929    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759932    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759935    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759938    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759941    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759943    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759946    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759948    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759951    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759954    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759956    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 11:29:53.763970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.759959    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760032    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760071    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760080    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760084    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760095    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760098    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760102    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760107    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760110    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760113    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760117    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760121    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760124    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760127    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760130    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760133    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760136    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760139    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760145    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760148    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760151    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760154    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760158    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 11:29:53.764508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760161    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760164    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760167    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760171    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760174    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760177    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760179    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760183    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760185    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760189    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760192    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760195    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760198    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760201    2561 flags.go:64] FLAG: --enable-server="true"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760207    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760214    2561 flags.go:64] FLAG: --event-burst="100"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760217    2561 flags.go:64] FLAG: --event-qps="50"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760219    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760223    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760226    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760229    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760232    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760235    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760238    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760242    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 11:29:53.765073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760245    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760248    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760251    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760254    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760257    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760260    2561 flags.go:64] FLAG: --feature-gates=""
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760264    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760266    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760270    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760273    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760275    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760278    2561 flags.go:64] FLAG: --help="false"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760281    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-139-41.ec2.internal"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760284    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760287    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760290    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760293    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760296    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760299    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760302    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760305    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760307    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760316    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760319    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 11:29:53.765693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760322    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760325    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760327    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760330    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760333    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760336    2561 flags.go:64] FLAG: --lock-file=""
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760340    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760343    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760346    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760352    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760354    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760357    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760360    2561 flags.go:64] FLAG: --logging-format="text"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760363    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760367    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760369    2561 flags.go:64] FLAG: --manifest-url=""
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760372    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760377    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760379    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760383    2561 flags.go:64] FLAG: --max-pods="110"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760386    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760389    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760392    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760395    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760397    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 11:29:53.766274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760400    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760403    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760410    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760413    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760416    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760424    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760433    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760439    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760441    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760444    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760447    2561 flags.go:64] FLAG: --port="10250"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760450    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760453    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e686576085f41e15"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760457    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760460    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760475    2561 flags.go:64] FLAG: --register-node="true"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760478    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760484    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760487    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760490    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760493    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760497    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760500    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760503    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760505    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 11:29:53.766898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760508    2561 flags.go:64] FLAG: --runonce="false"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760511    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760514    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760517    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760520    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760522    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760525    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760528    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760531    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760534    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760537    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760541    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760544    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760547    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760550    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760553    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760558    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760561    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760564    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760568    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760572    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760575    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760578    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760581    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760584    2561 flags.go:64] FLAG: --v="2"
Sep 08 11:29:53.767535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760587    2561 flags.go:64] FLAG: --version="false"
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760591    2561 flags.go:64] FLAG: --vmodule=""
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760595    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.760598    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760735    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760739    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760742    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760745    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760748    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760750    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760753    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760755    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760758    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760761    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760764    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760766    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760769    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760772    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760776    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760779    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760781    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 11:29:53.768176 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760784    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760787    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760789    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760792    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760794    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760797    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760801    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760803    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760806    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760809    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760811    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760814    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760816    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760820    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760824    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760827    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760829    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760832    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760834    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760837    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 11:29:53.768716 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760840    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760842    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760845    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760847    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760850    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760852    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760855    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760858    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760860    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760864    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760867    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760869    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760872    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760874    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760878    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760880    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760883    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760885    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760890    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760892    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 11:29:53.769193 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760895    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760898    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760900    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760903    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760905    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760908    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760910    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760913    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760915    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760918    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760921    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760923    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760926    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760936    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760940    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760944    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760947    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760950    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760953    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 11:29:53.769698 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760956    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760959    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760963    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760966    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760968    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760971    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760974    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760976    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760979    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760982    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760985    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760988    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760991    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760994    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760997    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.760999    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.761002    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.761004    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.761007    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 11:29:53.770172 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.761010    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.761013    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.761017    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.761020    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.761025    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.768619    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.768634    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.769563    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.769572    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769630    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769635    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769638    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769643    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769646    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769649    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 11:29:53.770657 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769652    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769656    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769659    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769662    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769665    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769668    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769671    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769674    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769677    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769679    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769682    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769684    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769687    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769690    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769693    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769695    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769698    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769701    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769703    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769706    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 11:29:53.771023 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769709    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769711    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769714    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769719    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769722    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769725    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769727    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769730    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769733    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769736    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769738    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769741    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769743    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769746    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769749    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769752    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769755    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769757    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769759    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769762    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 11:29:53.771571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769764    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769767    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769770    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769772    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769775    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769778    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769780    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769783    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769787    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769790    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769793    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769797    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769800    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769803    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769806    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769810    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769813    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769816    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769818    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769821    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 11:29:53.772069 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769824    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769826    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769829    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769832    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769834    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769837    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769840    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769844    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769847    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769851    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769855    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769858    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769861    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769863    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769866    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769869    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769871    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769874    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769876    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 11:29:53.772571 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769879    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769882    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769884    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769887    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769889    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769892    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769894    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769898    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769901    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769903    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769907    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769909    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769912    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.769914    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 11:29:53.773033 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.769920    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770034    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770039    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770042    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770044    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770047    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770050    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770053    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770056    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770058    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770061    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770064    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770068    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770071    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770074    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770077    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770080    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770082    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770085    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 11:29:53.773385 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770087    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770090    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770093    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770095    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770097    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770101    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770103    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770106    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770109    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770112    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770117    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770119    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770122    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770125    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770127    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770130    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770133    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770135    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770137    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 11:29:53.773856 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770140    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770142    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770145    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770148    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770150    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770152    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770155    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770157    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770160    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770163    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770165    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770168    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770170    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770173    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770175    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770178    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770181    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770183    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770187    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770189    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770192    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 11:29:53.774334 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770195    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770197    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770200    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770203    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770205    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770207    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770210    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770212    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770215    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770217    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770220    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770222    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770225    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770228    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770230    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770233    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770235    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770238    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770240    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770243    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 11:29:53.774970 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770245    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770249    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770252    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770255    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770258    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770261    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770264    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770266    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770269    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770272    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770275    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770278    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770281    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770284    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770287    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770289    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770292    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770294    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770297    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 11:29:53.775458 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770299    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 11:29:53.775936 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:53.770302    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 11:29:53.775936 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.770307    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 11:29:53.775936 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.770418    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 11:29:53.775936 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.772807    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 11:29:53.775936 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.773891    2561 server.go:1019] "Starting client certificate rotation"
Sep 08 11:29:53.775936 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.773989    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 11:29:53.775936 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.774014    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 11:29:53.779252 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.779236    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 11:29:53.783174 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.783149    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 11:29:53.790610 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.790587    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 08 11:29:53.795036 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.795016    2561 log.go:25] "Validated CRI v1 image API"
Sep 08 11:29:53.796073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.796041    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 11:29:53.799300 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.799279    2561 fs.go:135] Filesystem UUIDs: map[2c2b7691-68ca-423f-adea-275130ee262a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f4139477-0640-458f-b751-fb6e1d9a1d19:/dev/nvme0n1p4]
Sep 08 11:29:53.799384 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.799298    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 11:29:53.805478 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.804414    2561 manager.go:216] Machine: {Timestamp:2026-09-08 11:29:53.803037663 +0000 UTC m=+0.369798195 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105212 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec208ac26ef4c3d19da77ee33452bef6 SystemUUID:ec208ac2-6ef4-c3d1-9da7-7ee33452bef6 BootID:11305d7c-476c-4372-a382-4b2a6e277e36 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:23:cb:06:78:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:23:cb:06:78:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:4b:9f:58:4b:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 11:29:53.805478 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.805454    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 11:29:53.805604 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.805556    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 11:29:53.807099 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.807080    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 11:29:53.809205 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.809177    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 11:29:53.809344 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.809207    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-41.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 11:29:53.810785 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.810768    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 11:29:53.810785 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.810782    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 11:29:53.810899 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.810794    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 11:29:53.811684 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.811673    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 11:29:53.813661 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.813650    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 11:29:53.818446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.818435    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 11:29:53.818577 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.818567    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 11:29:53.818614 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.818610    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 11:29:53.818648 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.818627    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 11:29:53.818648 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.818638    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 11:29:53.818648 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.818648    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 08 11:29:53.818734 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.818658    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 11:29:53.819000 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.818987    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 11:29:53.819073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.819006    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 11:29:53.821380 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.821361    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 11:29:53.824388 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.824373    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 11:29:53.824451 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.824413    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 11:29:53.825357 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.825345    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 11:29:53.825396 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.825367    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 11:29:53.825396 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.825376    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 11:29:53.825396 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.825384    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 11:29:53.825498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.825397    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 11:29:53.825498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.825406    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 11:29:53.825498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.825424    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 11:29:53.825498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.825432    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 11:29:53.825498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.825448    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 11:29:53.825498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.825456    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 11:29:53.825498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.825483    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 11:29:53.825498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.825494    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 11:29:53.827411 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.827398    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 11:29:53.827411 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.827413    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 11:29:53.831978 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.831953    2561 server.go:1268] "Started kubelet"
Sep 08 11:29:53.832056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.831986    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-41.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 11:29:53.832056 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.831982    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 11:29:53.832056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.832036    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 11:29:53.832181 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.832045    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 11:29:53.835888 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.832082    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 11:29:53.835980 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.835948    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-41.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 11:29:53.836927 ip-10-0-139-41 systemd[1]: Started Kubernetes Kubelet.
Sep 08 11:29:53.839064 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.839046    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 11:29:53.839712 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.839690    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 11:29:53.842555 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.842527    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 11:29:53.844319 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.843181    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-41.ec2.internal.18d3550d95c5d985  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-41.ec2.internal,UID:ip-10-0-139-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-41.ec2.internal,},FirstTimestamp:2026-09-08 11:29:53.831926149 +0000 UTC m=+0.398686688,LastTimestamp:2026-09-08 11:29:53.831926149 +0000 UTC m=+0.398686688,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-41.ec2.internal,}"
Sep 08 11:29:53.846087 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.846068    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 11:29:53.846854 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.846835    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 11:29:53.846985 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.846968    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:53.847671 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.847550    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 11:29:53.847671 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.847551    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 11:29:53.847671 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.847577    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 11:29:53.847671 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.847641    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 11:29:53.847671 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.847649    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 11:29:53.848048 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.847692    2561 factory.go:55] Registering systemd factory
Sep 08 11:29:53.848048 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.847708    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 08 11:29:53.848236 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.848172    2561 factory.go:153] Registering CRI-O factory
Sep 08 11:29:53.848236 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.848186    2561 factory.go:223] Registration of the crio container factory successfully
Sep 08 11:29:53.848340 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.848238    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 11:29:53.848340 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.848258    2561 factory.go:103] Registering Raw factory
Sep 08 11:29:53.848340 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.848275    2561 manager.go:1196] Started watching for new ooms in manager
Sep 08 11:29:53.848965 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.848947    2561 manager.go:349] Starting recovery of all containers
Sep 08 11:29:53.850427 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.850398    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-41.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 11:29:53.850557 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.850536    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 11:29:53.859056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.859038    2561 manager.go:354] Recovery completed
Sep 08 11:29:53.863500 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.863487    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 11:29:53.865527 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.865509    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 11:29:53.865604 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.865543    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 11:29:53.865604 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.865558    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Sep 08 11:29:53.866141 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.866126    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 11:29:53.866141 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.866137    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 11:29:53.866141 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.866141    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 11:29:53.866288 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.866156    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 11:29:53.866865 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.866850    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 11:29:53.866908 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.866875    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 11:29:53.866908 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.866887    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Sep 08 11:29:53.867919 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.867855    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-41.ec2.internal.18d3550d97c68e9d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-41.ec2.internal,UID:ip-10-0-139-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-41.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-41.ec2.internal,},FirstTimestamp:2026-09-08 11:29:53.865526941 +0000 UTC m=+0.432287473,LastTimestamp:2026-09-08 11:29:53.865526941 +0000 UTC m=+0.432287473,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-41.ec2.internal,}"
Sep 08 11:29:53.868330 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.868314    2561 policy_none.go:49] "None policy: Start"
Sep 08 11:29:53.868386 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.868337    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 11:29:53.868386 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.868353    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 11:29:53.870439 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.870428    2561 policy_none.go:47] "Start"
Sep 08 11:29:53.877279 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.877189    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-41.ec2.internal.18d3550d97c6e966  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-41.ec2.internal,UID:ip-10-0-139-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-41.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-41.ec2.internal,},FirstTimestamp:2026-09-08 11:29:53.865550182 +0000 UTC m=+0.432310719,LastTimestamp:2026-09-08 11:29:53.865550182 +0000 UTC m=+0.432310719,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-41.ec2.internal,}"
Sep 08 11:29:53.885094 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.885027    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-41.ec2.internal.18d3550d97c71ebf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-41.ec2.internal,UID:ip-10-0-139-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-41.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-41.ec2.internal,},FirstTimestamp:2026-09-08 11:29:53.865563839 +0000 UTC m=+0.432324376,LastTimestamp:2026-09-08 11:29:53.865563839 +0000 UTC m=+0.432324376,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-41.ec2.internal,}"
Sep 08 11:29:53.893341 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.893323    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rq89l"
Sep 08 11:29:53.897490 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.897415    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-41.ec2.internal.18d3550d97c68e9d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-41.ec2.internal.18d3550d97c68e9d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-41.ec2.internal,UID:ip-10-0-139-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-41.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-41.ec2.internal,},FirstTimestamp:2026-09-08 11:29:53.865526941 +0000 UTC m=+0.432287473,LastTimestamp:2026-09-08 11:29:53.866863123 +0000 UTC m=+0.433623655,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-41.ec2.internal,}"
Sep 08 11:29:53.904963 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.904942    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rq89l"
Sep 08 11:29:53.928845 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.906403    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-41.ec2.internal.18d3550d97c6e966\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-41.ec2.internal.18d3550d97c6e966  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-41.ec2.internal,UID:ip-10-0-139-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-41.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-41.ec2.internal,},FirstTimestamp:2026-09-08 11:29:53.865550182 +0000 UTC m=+0.432310719,LastTimestamp:2026-09-08 11:29:53.86688235 +0000 UTC m=+0.433642883,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-41.ec2.internal,}"
Sep 08 11:29:53.928845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.911100    2561 manager.go:337] "Starting Device Plugin manager"
Sep 08 11:29:53.928845 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.911137    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 11:29:53.928845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.911146    2561 server.go:87] "Starting device plugin registration server"
Sep 08 11:29:53.928845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.911457    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 11:29:53.928845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.911486    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 11:29:53.928845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.911579    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 11:29:53.928845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.911669    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 11:29:53.928845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.911679    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 11:29:53.928845 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.912225    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 11:29:53.928845 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.912278    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:53.953235 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.953201    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 11:29:53.954407 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.954389    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 11:29:53.954523 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.954437    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 11:29:53.954523 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.954459    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 11:29:53.954618 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:53.954572    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 11:29:53.957248 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:53.957231    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 11:29:54.012354 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.012253    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 11:29:54.013232 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.013215    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 11:29:54.013328 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.013251    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 11:29:54.013328 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.013267    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Sep 08 11:29:54.013328 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.013299    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.022711 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.022694    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.022779 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:54.022716    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-41.ec2.internal\": node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:54.037395 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:54.037373    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:54.055191 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.055163    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"]
Sep 08 11:29:54.055292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.055235    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 11:29:54.056719 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.056702    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 11:29:54.056811 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.056731    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 11:29:54.056811 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.056740    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Sep 08 11:29:54.059253 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.059237    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 11:29:54.059372 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.059356    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.059441 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.059407    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 11:29:54.060008 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.059990    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 11:29:54.060073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.060022    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 11:29:54.060073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.060032    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Sep 08 11:29:54.060176 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.059994    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 11:29:54.060176 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.060093    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 11:29:54.060176 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.060107    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Sep 08 11:29:54.062376 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.062363    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.062435 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.062387    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 11:29:54.063081 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.063064    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 11:29:54.063156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.063094    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 11:29:54.063156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.063107    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Sep 08 11:29:54.097187 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:54.097164    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-41.ec2.internal\" not found" node="ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.101557 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:54.101540    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-41.ec2.internal\" not found" node="ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.137965 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:54.137943    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:54.149645 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.149618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9cb4470e82b2f2f8fdaaff0351211490-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"9cb4470e82b2f2f8fdaaff0351211490\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.149645 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.149645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cb4470e82b2f2f8fdaaff0351211490-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"9cb4470e82b2f2f8fdaaff0351211490\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.149764 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.149665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/15e2f71567686cf0984f394af0a6f5d0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-41.ec2.internal\" (UID: \"15e2f71567686cf0984f394af0a6f5d0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.238531 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:54.238490    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:54.249778 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.249752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9cb4470e82b2f2f8fdaaff0351211490-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"9cb4470e82b2f2f8fdaaff0351211490\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.249842 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.249782    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cb4470e82b2f2f8fdaaff0351211490-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"9cb4470e82b2f2f8fdaaff0351211490\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.249842 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.249799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/15e2f71567686cf0984f394af0a6f5d0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-41.ec2.internal\" (UID: \"15e2f71567686cf0984f394af0a6f5d0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.249908 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.249848    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cb4470e82b2f2f8fdaaff0351211490-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"9cb4470e82b2f2f8fdaaff0351211490\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.249908 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.249854    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9cb4470e82b2f2f8fdaaff0351211490-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal\" (UID: \"9cb4470e82b2f2f8fdaaff0351211490\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.249908 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.249894    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/15e2f71567686cf0984f394af0a6f5d0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-41.ec2.internal\" (UID: \"15e2f71567686cf0984f394af0a6f5d0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.339292 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:54.339200    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:54.400751 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.400705    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.404078 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.404060    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:54.440340 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:54.440309    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:54.546453 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:54.546418    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:54.654143 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:54.654068    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:54.754857 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:54.754823    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:54.774454 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.774432    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 11:29:54.774615 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.774595    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 11:29:54.855676 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:54.855648    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:54.908078 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.907990    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 11:24:54 +0000 UTC" deadline="2028-03-29 21:21:07.690505867 +0000 UTC"
Sep 08 11:29:54.908078 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.908029    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13641h51m12.782480857s"
Sep 08 11:29:54.956113 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:54.956079    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:54.970462 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:54.970426    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 11:29:55.051760 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:55.051727    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e2f71567686cf0984f394af0a6f5d0.slice/crio-d08c89818919a078553c4cfe3a2c9341e9f0586fccca3a47d9b23ca57677bb13 WatchSource:0}: Error finding container d08c89818919a078553c4cfe3a2c9341e9f0586fccca3a47d9b23ca57677bb13: Status 404 returned error can't find the container with id d08c89818919a078553c4cfe3a2c9341e9f0586fccca3a47d9b23ca57677bb13
Sep 08 11:29:55.051981 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:55.051959    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb4470e82b2f2f8fdaaff0351211490.slice/crio-2c8697a7348b7ef8a8003b1b4d4e2e603e222c05e8b31e15fcb178144645236e WatchSource:0}: Error finding container 2c8697a7348b7ef8a8003b1b4d4e2e603e222c05e8b31e15fcb178144645236e: Status 404 returned error can't find the container with id 2c8697a7348b7ef8a8003b1b4d4e2e603e222c05e8b31e15fcb178144645236e
Sep 08 11:29:55.056469 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:55.056448    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:55.056566 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.056548    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 11:29:55.062838 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.062821    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 11:29:55.063627 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.063612    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 11:29:55.063689 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.063639    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 11:29:55.063689 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.063650    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeHasSufficientPID"
Sep 08 11:29:55.157603 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:55.157559    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:55.229115 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.229043    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 11:29:55.258620 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:55.258593    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:55.358881 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:55.358841    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-41.ec2.internal\" not found"
Sep 08 11:29:55.408719 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.408691    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 11:29:55.447328 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.447295    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:55.462178 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.462149    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 11:29:55.463061 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.463046    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal"
Sep 08 11:29:55.471919 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.471900    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 11:29:55.820217 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.820175    2561 apiserver.go:52] "Watching apiserver"
Sep 08 11:29:55.830690 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.830657    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 11:29:55.831114 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.831086    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-66cjb","openshift-multus/multus-ltd4c","openshift-network-operator/iptables-alerter-56s5x","kube-system/global-pull-secret-syncer-p9x77","kube-system/konnectivity-agent-4w4xp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal","openshift-multus/network-metrics-daemon-q2z6p","openshift-network-diagnostics/network-check-target-zgxcv","openshift-ovn-kubernetes/ovnkube-node-v4wfz","kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5nbml","openshift-dns/node-resolver-l76h7","openshift-image-registry/node-ca-5xw68"]
Sep 08 11:29:55.834104 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.834073    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.836524 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.836502    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l76h7"
Sep 08 11:29:55.837150 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.837128    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 11:29:55.837253 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.837229    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 11:29:55.837322 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.837288    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 11:29:55.837379 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.837329    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 11:29:55.837379 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.837130    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hjb96\""
Sep 08 11:29:55.837479 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.837447    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 11:29:55.838330 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.838305    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 11:29:55.838980 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.838959    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-56s5x"
Sep 08 11:29:55.841186 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.841164    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 11:29:55.841478 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.841459    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:55.841605 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:55.841563    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:29:55.841990 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.841970    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 11:29:55.842519 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.842350    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5f2gd\""
Sep 08 11:29:55.844052 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.843344    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 11:29:55.844052 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.843587    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tkm2n\""
Sep 08 11:29:55.844052 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.843900    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 11:29:55.844052 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.843956    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 11:29:55.844783 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.844324    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4w4xp"
Sep 08 11:29:55.846478 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.846459    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 11:29:55.846705 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.846686    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 11:29:55.846988 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.846968    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.847183 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.847165    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 11:29:55.847600 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.847532    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wcm6b\""
Sep 08 11:29:55.850306 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.850287    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:29:55.850410 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:55.850353    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:29:55.850410 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.850380    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 11:29:55.851083 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.850865    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 11:29:55.851083 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.850957    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fdzwn\""
Sep 08 11:29:55.851227 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.851200    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 11:29:55.852826 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.852807    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5xw68"
Sep 08 11:29:55.855327 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.855301    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.856196 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.856067    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 11:29:55.856488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.856452    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 11:29:55.856854 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.856837    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 11:29:55.857527 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.857504    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8lvj4\""
Sep 08 11:29:55.858378 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.858261    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 11:29:55.858582 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.858560    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 11:29:55.858664 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.858598    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-md8wx\""
Sep 08 11:29:55.858941 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.858844    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 11:29:55.859074 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.859024    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 11:29:55.859627 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.859402    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:55.862016 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.861994    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 11:29:55.862503 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862435    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-run-ovn\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.862503 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00567f83-0336-467f-826b-a57e063c76ac-tmp-dir\") pod \"node-resolver-l76h7\" (UID: \"00567f83-0336-467f-826b-a57e063c76ac\") " pod="openshift-dns/node-resolver-l76h7"
Sep 08 11:29:55.862656 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw7hp\" (UniqueName: \"kubernetes.io/projected/00567f83-0336-467f-826b-a57e063c76ac-kube-api-access-zw7hp\") pod \"node-resolver-l76h7\" (UID: \"00567f83-0336-467f-826b-a57e063c76ac\") " pod="openshift-dns/node-resolver-l76h7"
Sep 08 11:29:55.862656 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3cac5fa2-8f4b-4fdf-89c6-353e3e317e01-host-slash\") pod \"iptables-alerter-56s5x\" (UID: \"3cac5fa2-8f4b-4fdf-89c6-353e3e317e01\") " pod="openshift-network-operator/iptables-alerter-56s5x"
Sep 08 11:29:55.862656 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862594    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0dff2c8b-ae64-424e-9b34-28819a39597d-dbus\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:55.862656 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862623    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-socket-dir\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.862852 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-var-lib-openvswitch\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.862852 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3cac5fa2-8f4b-4fdf-89c6-353e3e317e01-iptables-alerter-script\") pod \"iptables-alerter-56s5x\" (UID: \"3cac5fa2-8f4b-4fdf-89c6-353e3e317e01\") " pod="openshift-network-operator/iptables-alerter-56s5x"
Sep 08 11:29:55.862852 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcxdj\" (UniqueName: \"kubernetes.io/projected/3cac5fa2-8f4b-4fdf-89c6-353e3e317e01-kube-api-access-mcxdj\") pod \"iptables-alerter-56s5x\" (UID: \"3cac5fa2-8f4b-4fdf-89c6-353e3e317e01\") " pod="openshift-network-operator/iptables-alerter-56s5x"
Sep 08 11:29:55.862852 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50cf80de-8fa6-4300-a09e-4a95c55ef193-konnectivity-ca\") pod \"konnectivity-agent-4w4xp\" (UID: \"50cf80de-8fa6-4300-a09e-4a95c55ef193\") " pod="kube-system/konnectivity-agent-4w4xp"
Sep 08 11:29:55.862852 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862777    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-registration-dir\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.862852 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862801    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-sys-fs\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.862852 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-kubelet\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.863196 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-slash\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.863196 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862920    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.863309 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.862948    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.863368 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.863342    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0d6b121-1b5d-4967-860a-c30e742876bf-env-overrides\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.863427 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.863387    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz56d\" (UniqueName: \"kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d\") pod \"network-check-target-zgxcv\" (UID: \"7849d5bd-5ec1-4d67-b508-b361e4f347d0\") " pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:29:55.863890 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.863828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-run-systemd\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.863974 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.863883    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-log-socket\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.863974 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.863935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtndl\" (UniqueName: \"kubernetes.io/projected/d0d6b121-1b5d-4967-860a-c30e742876bf-kube-api-access-xtndl\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.863974 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.863966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.864140 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.864001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-etc-openvswitch\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.864977 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.864957    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:29:55.865097 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:55.865068    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:29:55.865467 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.865445    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 11:29:55.866683 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.866662    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 11:29:55.866805 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.866784    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qkrr8\""
Sep 08 11:29:55.867029 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.867006    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-node-log\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.867093 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.867064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0d6b121-1b5d-4967-860a-c30e742876bf-ovnkube-config\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.867150 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.867111    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-run-openvswitch\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.867203 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.867148    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0d6b121-1b5d-4967-860a-c30e742876bf-ovn-node-metrics-cert\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.867203 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.867192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0dff2c8b-ae64-424e-9b34-28819a39597d-kubelet-config\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:55.867313 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.867230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:55.867313 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.867296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-etc-selinux\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.867835 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.867466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgcrm\" (UniqueName: \"kubernetes.io/projected/02895789-6e33-4cf0-8ebb-d61bbea32d37-kube-api-access-pgcrm\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.868008 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.867919    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-cni-netd\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.868008 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.867975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00567f83-0336-467f-826b-a57e063c76ac-hosts-file\") pod \"node-resolver-l76h7\" (UID: \"00567f83-0336-467f-826b-a57e063c76ac\") " pod="openshift-dns/node-resolver-l76h7"
Sep 08 11:29:55.868120 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.868014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50cf80de-8fa6-4300-a09e-4a95c55ef193-agent-certs\") pod \"konnectivity-agent-4w4xp\" (UID: \"50cf80de-8fa6-4300-a09e-4a95c55ef193\") " pod="kube-system/konnectivity-agent-4w4xp"
Sep 08 11:29:55.868230 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.868211    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-systemd-units\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.868283 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.868250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-run-netns\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.868344 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.868282    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-cni-bin\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.868399 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.868339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0d6b121-1b5d-4967-860a-c30e742876bf-ovnkube-script-lib\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.868399 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.868371    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-device-dir\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.870234 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.870214    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.873927 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.873380    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jsb4t\""
Sep 08 11:29:55.873927 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.873385    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 11:29:55.873927 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.873395    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 11:29:55.894740 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.894615    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kxlfx"
Sep 08 11:29:55.904719 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.904696    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kxlfx"
Sep 08 11:29:55.949213 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.949185    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 11:29:55.961131 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.961051    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" event={"ID":"15e2f71567686cf0984f394af0a6f5d0","Type":"ContainerStarted","Data":"d08c89818919a078553c4cfe3a2c9341e9f0586fccca3a47d9b23ca57677bb13"}
Sep 08 11:29:55.963334 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.963293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" event={"ID":"9cb4470e82b2f2f8fdaaff0351211490","Type":"ContainerStarted","Data":"2c8697a7348b7ef8a8003b1b4d4e2e603e222c05e8b31e15fcb178144645236e"}
Sep 08 11:29:55.968599 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-run-netns\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.968713 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-run-multus-certs\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.968713 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968637    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00567f83-0336-467f-826b-a57e063c76ac-hosts-file\") pod \"node-resolver-l76h7\" (UID: \"00567f83-0336-467f-826b-a57e063c76ac\") " pod="openshift-dns/node-resolver-l76h7"
Sep 08 11:29:55.968713 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968661    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85a2c614-d7f0-4e48-a229-ede402992c01-multus-daemon-config\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.968713 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h55pc\" (UniqueName: \"kubernetes.io/projected/85a2c614-d7f0-4e48-a229-ede402992c01-kube-api-access-h55pc\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.968713 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-systemd-units\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.968989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968739    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-run-netns\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.968989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-cni-bin\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.968989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-device-dir\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.968989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-run-netns\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.968989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-kubernetes\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.968989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968854    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-systemd-units\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.968989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968862    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-sysctl-d\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.968989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968738    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00567f83-0336-467f-826b-a57e063c76ac-hosts-file\") pod \"node-resolver-l76h7\" (UID: \"00567f83-0336-467f-826b-a57e063c76ac\") " pod="openshift-dns/node-resolver-l76h7"
Sep 08 11:29:55.968989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968889    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-run-ovn\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.968989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968910    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-cni-bin\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.968989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968914    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00567f83-0336-467f-826b-a57e063c76ac-tmp-dir\") pod \"node-resolver-l76h7\" (UID: \"00567f83-0336-467f-826b-a57e063c76ac\") " pod="openshift-dns/node-resolver-l76h7"
Sep 08 11:29:55.968989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0dff2c8b-ae64-424e-9b34-28819a39597d-dbus\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:55.968989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.968989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-socket-dir\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969002    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-run-ovn\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl69v\" (UniqueName: \"kubernetes.io/projected/3a72df4f-cfd6-4867-92ee-7f559401f9d0-kube-api-access-fl69v\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-sysctl-conf\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-lib-modules\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969077    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-var-lib-openvswitch\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969083    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0dff2c8b-ae64-424e-9b34-28819a39597d-dbus\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969103    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcxdj\" (UniqueName: \"kubernetes.io/projected/3cac5fa2-8f4b-4fdf-89c6-353e3e317e01-kube-api-access-mcxdj\") pod \"iptables-alerter-56s5x\" (UID: \"3cac5fa2-8f4b-4fdf-89c6-353e3e317e01\") " pod="openshift-network-operator/iptables-alerter-56s5x"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969135    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50cf80de-8fa6-4300-a09e-4a95c55ef193-konnectivity-ca\") pod \"konnectivity-agent-4w4xp\" (UID: \"50cf80de-8fa6-4300-a09e-4a95c55ef193\") " pod="kube-system/konnectivity-agent-4w4xp"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969159    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-sys-fs\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969179    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/838a5800-8a91-4b1b-bc39-655796992d82-host\") pod \"node-ca-5xw68\" (UID: \"838a5800-8a91-4b1b-bc39-655796992d82\") " pod="openshift-image-registry/node-ca-5xw68"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969211    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/838a5800-8a91-4b1b-bc39-655796992d82-serviceca\") pod \"node-ca-5xw68\" (UID: \"838a5800-8a91-4b1b-bc39-655796992d82\") " pod="openshift-image-registry/node-ca-5xw68"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969248    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-systemd\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-kubelet\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-slash\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969358    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0d6b121-1b5d-4967-860a-c30e742876bf-env-overrides\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.969688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz56d\" (UniqueName: \"kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d\") pod \"network-check-target-zgxcv\" (UID: \"7849d5bd-5ec1-4d67-b508-b361e4f347d0\") " pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969411    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-sys-fs\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00567f83-0336-467f-826b-a57e063c76ac-tmp-dir\") pod \"node-resolver-l76h7\" (UID: \"00567f83-0336-467f-826b-a57e063c76ac\") " pod="openshift-dns/node-resolver-l76h7"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-device-dir\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969432    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85a2c614-d7f0-4e48-a229-ede402992c01-cni-binary-copy\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969614    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-kubelet\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-slash\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-var-lib-cni-bin\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969670    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-var-lib-openvswitch\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969711    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-modprobe-d\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969778    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtndl\" (UniqueName: \"kubernetes.io/projected/d0d6b121-1b5d-4967-860a-c30e742876bf-kube-api-access-xtndl\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-multus-cni-dir\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969827    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-socket-dir\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-multus-socket-dir-parent\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-cnibin\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969895    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-sys\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.970488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969922    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-etc-openvswitch\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969986    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-etc-openvswitch\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.969988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-node-log\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-node-log\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0d6b121-1b5d-4967-860a-c30e742876bf-ovnkube-config\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970071    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-run-k8s-cni-cncf-io\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970123    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-var-lib-kubelet\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970127    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0d6b121-1b5d-4967-860a-c30e742876bf-env-overrides\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-host\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhwfq\" (UniqueName: \"kubernetes.io/projected/0ad64a4b-1208-4583-a7cb-4f047fb46203-kube-api-access-hhwfq\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgcrm\" (UniqueName: \"kubernetes.io/projected/02895789-6e33-4cf0-8ebb-d61bbea32d37-kube-api-access-pgcrm\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970440    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-etc-kubernetes\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970491    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-sysconfig\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970521    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-tuned\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970565    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-cni-netd\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970600    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50cf80de-8fa6-4300-a09e-4a95c55ef193-agent-certs\") pod \"konnectivity-agent-4w4xp\" (UID: \"50cf80de-8fa6-4300-a09e-4a95c55ef193\") " pod="kube-system/konnectivity-agent-4w4xp"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970608    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0d6b121-1b5d-4967-860a-c30e742876bf-ovnkube-config\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.971397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970631    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-system-cni-dir\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-cni-netd\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970657    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-hostroot\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0d6b121-1b5d-4967-860a-c30e742876bf-ovnkube-script-lib\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970744    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50cf80de-8fa6-4300-a09e-4a95c55ef193-konnectivity-ca\") pod \"konnectivity-agent-4w4xp\" (UID: \"50cf80de-8fa6-4300-a09e-4a95c55ef193\") " pod="kube-system/konnectivity-agent-4w4xp"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970908    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw7hp\" (UniqueName: \"kubernetes.io/projected/00567f83-0336-467f-826b-a57e063c76ac-kube-api-access-zw7hp\") pod \"node-resolver-l76h7\" (UID: \"00567f83-0336-467f-826b-a57e063c76ac\") " pod="openshift-dns/node-resolver-l76h7"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.970972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3cac5fa2-8f4b-4fdf-89c6-353e3e317e01-host-slash\") pod \"iptables-alerter-56s5x\" (UID: \"3cac5fa2-8f4b-4fdf-89c6-353e3e317e01\") " pod="openshift-network-operator/iptables-alerter-56s5x"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-system-cni-dir\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971041    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3cac5fa2-8f4b-4fdf-89c6-353e3e317e01-iptables-alerter-script\") pod \"iptables-alerter-56s5x\" (UID: \"3cac5fa2-8f4b-4fdf-89c6-353e3e317e01\") " pod="openshift-network-operator/iptables-alerter-56s5x"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971091    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-registration-dir\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-cni-binary-copy\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971140    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-var-lib-kubelet\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad64a4b-1208-4583-a7cb-4f047fb46203-tmp\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.972292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbtbh\" (UniqueName: \"kubernetes.io/projected/838a5800-8a91-4b1b-bc39-655796992d82-kube-api-access-qbtbh\") pod \"node-ca-5xw68\" (UID: \"838a5800-8a91-4b1b-bc39-655796992d82\") " pod="openshift-image-registry/node-ca-5xw68"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971255    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-registration-dir\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0d6b121-1b5d-4967-860a-c30e742876bf-ovnkube-script-lib\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971323    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3cac5fa2-8f4b-4fdf-89c6-353e3e317e01-host-slash\") pod \"iptables-alerter-56s5x\" (UID: \"3cac5fa2-8f4b-4fdf-89c6-353e3e317e01\") " pod="openshift-network-operator/iptables-alerter-56s5x"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971393    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-run-systemd\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-log-socket\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971522    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-run-systemd\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971562    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971586    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-var-lib-cni-multus\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971610    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-run\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-cnibin\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971745    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-multus-conf-dir\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971757    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-log-socket\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3cac5fa2-8f4b-4fdf-89c6-353e3e317e01-iptables-alerter-script\") pod \"iptables-alerter-56s5x\" (UID: \"3cac5fa2-8f4b-4fdf-89c6-353e3e317e01\") " pod="openshift-network-operator/iptables-alerter-56s5x"
Sep 08 11:29:55.973156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:55.973951 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkpsr\" (UniqueName: \"kubernetes.io/projected/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-kube-api-access-jkpsr\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:55.973951 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-run-openvswitch\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.973951 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0d6b121-1b5d-4967-860a-c30e742876bf-ovn-node-metrics-cert\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.973951 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0dff2c8b-ae64-424e-9b34-28819a39597d-kubelet-config\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:55.973951 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971916    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:55.973951 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-etc-selinux\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.973951 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.971942    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0d6b121-1b5d-4967-860a-c30e742876bf-run-openvswitch\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.973951 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.972034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02895789-6e33-4cf0-8ebb-d61bbea32d37-etc-selinux\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.973951 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:55.972094    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 11:29:55.973951 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.972110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0dff2c8b-ae64-424e-9b34-28819a39597d-kubelet-config\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:55.973951 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:55.972258    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret podName:0dff2c8b-ae64-424e-9b34-28819a39597d nodeName:}" failed. No retries permitted until 2026-09-08 11:29:56.472226835 +0000 UTC m=+3.038987370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret") pod "global-pull-secret-syncer-p9x77" (UID: "0dff2c8b-ae64-424e-9b34-28819a39597d") : object "kube-system"/"original-pull-secret" not registered
Sep 08 11:29:55.976712 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.976681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0d6b121-1b5d-4967-860a-c30e742876bf-ovn-node-metrics-cert\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.976818 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.976738    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50cf80de-8fa6-4300-a09e-4a95c55ef193-agent-certs\") pod \"konnectivity-agent-4w4xp\" (UID: \"50cf80de-8fa6-4300-a09e-4a95c55ef193\") " pod="kube-system/konnectivity-agent-4w4xp"
Sep 08 11:29:55.982471 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.982450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtndl\" (UniqueName: \"kubernetes.io/projected/d0d6b121-1b5d-4967-860a-c30e742876bf-kube-api-access-xtndl\") pod \"ovnkube-node-v4wfz\" (UID: \"d0d6b121-1b5d-4967-860a-c30e742876bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:55.982882 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:55.982779    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 11:29:55.982882 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:55.982807    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 11:29:55.982882 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:55.982820    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qz56d for pod openshift-network-diagnostics/network-check-target-zgxcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:29:55.983072 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:55.982885    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d podName:7849d5bd-5ec1-4d67-b508-b361e4f347d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:29:56.482867467 +0000 UTC m=+3.049628023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qz56d" (UniqueName: "kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d") pod "network-check-target-zgxcv" (UID: "7849d5bd-5ec1-4d67-b508-b361e4f347d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:29:55.985668 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.985642    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcxdj\" (UniqueName: \"kubernetes.io/projected/3cac5fa2-8f4b-4fdf-89c6-353e3e317e01-kube-api-access-mcxdj\") pod \"iptables-alerter-56s5x\" (UID: \"3cac5fa2-8f4b-4fdf-89c6-353e3e317e01\") " pod="openshift-network-operator/iptables-alerter-56s5x"
Sep 08 11:29:55.986156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.986137    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgcrm\" (UniqueName: \"kubernetes.io/projected/02895789-6e33-4cf0-8ebb-d61bbea32d37-kube-api-access-pgcrm\") pod \"aws-ebs-csi-driver-node-c4klc\" (UID: \"02895789-6e33-4cf0-8ebb-d61bbea32d37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:55.988173 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:55.988148    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw7hp\" (UniqueName: \"kubernetes.io/projected/00567f83-0336-467f-826b-a57e063c76ac-kube-api-access-zw7hp\") pod \"node-resolver-l76h7\" (UID: \"00567f83-0336-467f-826b-a57e063c76ac\") " pod="openshift-dns/node-resolver-l76h7"
Sep 08 11:29:56.073164 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/838a5800-8a91-4b1b-bc39-655796992d82-host\") pod \"node-ca-5xw68\" (UID: \"838a5800-8a91-4b1b-bc39-655796992d82\") " pod="openshift-image-registry/node-ca-5xw68"
Sep 08 11:29:56.073164 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/838a5800-8a91-4b1b-bc39-655796992d82-serviceca\") pod \"node-ca-5xw68\" (UID: \"838a5800-8a91-4b1b-bc39-655796992d82\") " pod="openshift-image-registry/node-ca-5xw68"
Sep 08 11:29:56.073164 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073132    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-systemd\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073200    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/838a5800-8a91-4b1b-bc39-655796992d82-host\") pod \"node-ca-5xw68\" (UID: \"838a5800-8a91-4b1b-bc39-655796992d82\") " pod="openshift-image-registry/node-ca-5xw68"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85a2c614-d7f0-4e48-a229-ede402992c01-cni-binary-copy\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-var-lib-cni-bin\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-systemd\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073277    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-modprobe-d\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-multus-cni-dir\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073316    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-var-lib-cni-bin\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-multus-socket-dir-parent\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-cnibin\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-sys\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073375    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-multus-cni-dir\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073388    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-run-k8s-cni-cncf-io\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073388    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-modprobe-d\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073412    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-var-lib-kubelet\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073426    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-sys\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073435    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-host\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.073446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073436    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-cnibin\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-multus-socket-dir-parent\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-run-k8s-cni-cncf-io\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073459    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhwfq\" (UniqueName: \"kubernetes.io/projected/0ad64a4b-1208-4583-a7cb-4f047fb46203-kube-api-access-hhwfq\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-etc-kubernetes\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073539    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-sysconfig\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-tuned\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-sysconfig\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073632    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-system-cni-dir\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-etc-kubernetes\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073667    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-var-lib-kubelet\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-host\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/838a5800-8a91-4b1b-bc39-655796992d82-serviceca\") pod \"node-ca-5xw68\" (UID: \"838a5800-8a91-4b1b-bc39-655796992d82\") " pod="openshift-image-registry/node-ca-5xw68"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-system-cni-dir\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073706    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-hostroot\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-hostroot\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073778    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:29:56.074362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-system-cni-dir\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073835    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-cni-binary-copy\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85a2c614-d7f0-4e48-a229-ede402992c01-cni-binary-copy\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-var-lib-kubelet\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad64a4b-1208-4583-a7cb-4f047fb46203-tmp\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbtbh\" (UniqueName: \"kubernetes.io/projected/838a5800-8a91-4b1b-bc39-655796992d82-kube-api-access-qbtbh\") pod \"node-ca-5xw68\" (UID: \"838a5800-8a91-4b1b-bc39-655796992d82\") " pod="openshift-image-registry/node-ca-5xw68"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-system-cni-dir\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:56.073975    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073997    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-var-lib-kubelet\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074020    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-var-lib-cni-multus\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:56.074070    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs podName:3a72df4f-cfd6-4867-92ee-7f559401f9d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:29:56.574048552 +0000 UTC m=+3.140809092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs") pod "network-metrics-daemon-q2z6p" (UID: "3a72df4f-cfd6-4867-92ee-7f559401f9d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.073978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-var-lib-cni-multus\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-run\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-cnibin\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-multus-conf-dir\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-run\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.075185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkpsr\" (UniqueName: \"kubernetes.io/projected/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-kube-api-access-jkpsr\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-multus-conf-dir\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-cnibin\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074377    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-run-netns\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-run-multus-certs\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85a2c614-d7f0-4e48-a229-ede402992c01-multus-daemon-config\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h55pc\" (UniqueName: \"kubernetes.io/projected/85a2c614-d7f0-4e48-a229-ede402992c01-kube-api-access-h55pc\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074513    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-kubernetes\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074550    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-cni-binary-copy\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-run-multus-certs\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074558    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074604    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85a2c614-d7f0-4e48-a229-ede402992c01-host-run-netns\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074612    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-sysctl-d\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-kubernetes\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.075917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl69v\" (UniqueName: \"kubernetes.io/projected/3a72df4f-cfd6-4867-92ee-7f559401f9d0-kube-api-access-fl69v\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:29:56.076624 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-sysctl-conf\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.076624 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074700    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-lib-modules\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.076624 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-sysctl-d\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.076624 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074812    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-lib-modules\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.076624 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.074859    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-sysctl-conf\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.076624 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.075092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85a2c614-d7f0-4e48-a229-ede402992c01-multus-daemon-config\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.076624 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.076507    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ad64a4b-1208-4583-a7cb-4f047fb46203-etc-tuned\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.076988 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.076963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad64a4b-1208-4583-a7cb-4f047fb46203-tmp\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.091369 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.091340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkpsr\" (UniqueName: \"kubernetes.io/projected/a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d-kube-api-access-jkpsr\") pod \"multus-additional-cni-plugins-66cjb\" (UID: \"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d\") " pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.091498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.091395    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h55pc\" (UniqueName: \"kubernetes.io/projected/85a2c614-d7f0-4e48-a229-ede402992c01-kube-api-access-h55pc\") pod \"multus-ltd4c\" (UID: \"85a2c614-d7f0-4e48-a229-ede402992c01\") " pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.091498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.091455    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbtbh\" (UniqueName: \"kubernetes.io/projected/838a5800-8a91-4b1b-bc39-655796992d82-kube-api-access-qbtbh\") pod \"node-ca-5xw68\" (UID: \"838a5800-8a91-4b1b-bc39-655796992d82\") " pod="openshift-image-registry/node-ca-5xw68"
Sep 08 11:29:56.092072 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.092047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl69v\" (UniqueName: \"kubernetes.io/projected/3a72df4f-cfd6-4867-92ee-7f559401f9d0-kube-api-access-fl69v\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:29:56.092173 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.092101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhwfq\" (UniqueName: \"kubernetes.io/projected/0ad64a4b-1208-4583-a7cb-4f047fb46203-kube-api-access-hhwfq\") pod \"tuned-5nbml\" (UID: \"0ad64a4b-1208-4583-a7cb-4f047fb46203\") " pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.147236 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.147193    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:29:56.155955 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.155931    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l76h7"
Sep 08 11:29:56.172986 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.172951    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-56s5x"
Sep 08 11:29:56.175262 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.175234    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4w4xp"
Sep 08 11:29:56.184336 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.184313    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc"
Sep 08 11:29:56.190319 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.190298    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5xw68"
Sep 08 11:29:56.198233 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.198210    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ltd4c"
Sep 08 11:29:56.198329 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.198214    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 11:29:56.208028 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.208008    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-66cjb"
Sep 08 11:29:56.213825 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.213807    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5nbml"
Sep 08 11:29:56.477843 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.477748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:56.478001 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:56.477935    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 11:29:56.478063 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:56.478011    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret podName:0dff2c8b-ae64-424e-9b34-28819a39597d nodeName:}" failed. No retries permitted until 2026-09-08 11:29:57.477994667 +0000 UTC m=+4.044755211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret") pod "global-pull-secret-syncer-p9x77" (UID: "0dff2c8b-ae64-424e-9b34-28819a39597d") : object "kube-system"/"original-pull-secret" not registered
Sep 08 11:29:56.578994 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.578953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz56d\" (UniqueName: \"kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d\") pod \"network-check-target-zgxcv\" (UID: \"7849d5bd-5ec1-4d67-b508-b361e4f347d0\") " pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:29:56.579200 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.579011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:29:56.579200 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:56.579132    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:29:56.579200 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:56.579160    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 11:29:56.579200 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:56.579188    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 11:29:56.579200 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:56.579202    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs podName:3a72df4f-cfd6-4867-92ee-7f559401f9d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:29:57.579185387 +0000 UTC m=+4.145945928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs") pod "network-metrics-daemon-q2z6p" (UID: "3a72df4f-cfd6-4867-92ee-7f559401f9d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:29:56.579434 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:56.579203    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qz56d for pod openshift-network-diagnostics/network-check-target-zgxcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:29:56.579434 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:56.579264    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d podName:7849d5bd-5ec1-4d67-b508-b361e4f347d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:29:57.579245591 +0000 UTC m=+4.146006123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qz56d" (UniqueName: "kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d") pod "network-check-target-zgxcv" (UID: "7849d5bd-5ec1-4d67-b508-b361e4f347d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:29:56.908584 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.908498    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 11:24:56 +0000 UTC" deadline="2028-06-02 11:32:18.037161336 +0000 UTC"
Sep 08 11:29:56.908584 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.908532    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15192h2m21.128632745s"
Sep 08 11:29:56.955017 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.954975    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:56.955202 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:56.954975    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:29:56.955202 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:56.955133    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:29:56.955327 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:56.955229    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:29:57.092606 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:57.092319    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838a5800_8a91_4b1b_bc39_655796992d82.slice/crio-0172ac3bc6e54e4f740f22e7d86c63f41c2e4d82316d73a27ea2d2a26554bbc7 WatchSource:0}: Error finding container 0172ac3bc6e54e4f740f22e7d86c63f41c2e4d82316d73a27ea2d2a26554bbc7: Status 404 returned error can't find the container with id 0172ac3bc6e54e4f740f22e7d86c63f41c2e4d82316d73a27ea2d2a26554bbc7
Sep 08 11:29:57.094315 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:57.094295    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad64a4b_1208_4583_a7cb_4f047fb46203.slice/crio-9a985490e078fd0841bbc7cf9b49615fd9ad8764d97945f02cf55baedd0417c2 WatchSource:0}: Error finding container 9a985490e078fd0841bbc7cf9b49615fd9ad8764d97945f02cf55baedd0417c2: Status 404 returned error can't find the container with id 9a985490e078fd0841bbc7cf9b49615fd9ad8764d97945f02cf55baedd0417c2
Sep 08 11:29:57.097876 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:57.097834    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02895789_6e33_4cf0_8ebb_d61bbea32d37.slice/crio-7b06c81662038222fc6cda1ae01ab9c9d981414a23cfc94f5d73f9e3307e15d5 WatchSource:0}: Error finding container 7b06c81662038222fc6cda1ae01ab9c9d981414a23cfc94f5d73f9e3307e15d5: Status 404 returned error can't find the container with id 7b06c81662038222fc6cda1ae01ab9c9d981414a23cfc94f5d73f9e3307e15d5
Sep 08 11:29:57.099181 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:29:57.099157    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d6b121_1b5d_4967_860a_c30e742876bf.slice/crio-e10d729fee6fac365b462790f3e36fe40e7ee3fa0dd51a3947b442851af5128e WatchSource:0}: Error finding container e10d729fee6fac365b462790f3e36fe40e7ee3fa0dd51a3947b442851af5128e: Status 404 returned error can't find the container with id e10d729fee6fac365b462790f3e36fe40e7ee3fa0dd51a3947b442851af5128e
Sep 08 11:29:57.487424 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:57.487334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:57.487590 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:57.487473    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 11:29:57.487590 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:57.487559    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret podName:0dff2c8b-ae64-424e-9b34-28819a39597d nodeName:}" failed. No retries permitted until 2026-09-08 11:29:59.487537952 +0000 UTC m=+6.054298498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret") pod "global-pull-secret-syncer-p9x77" (UID: "0dff2c8b-ae64-424e-9b34-28819a39597d") : object "kube-system"/"original-pull-secret" not registered
Sep 08 11:29:57.589436 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:57.588156    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:29:57.589436 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:57.588272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz56d\" (UniqueName: \"kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d\") pod \"network-check-target-zgxcv\" (UID: \"7849d5bd-5ec1-4d67-b508-b361e4f347d0\") " pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:29:57.589436 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:57.588437    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 11:29:57.589436 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:57.588459    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 11:29:57.589436 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:57.588474    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qz56d for pod openshift-network-diagnostics/network-check-target-zgxcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:29:57.589436 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:57.588559    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d podName:7849d5bd-5ec1-4d67-b508-b361e4f347d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:29:59.588536578 +0000 UTC m=+6.155297121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qz56d" (UniqueName: "kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d") pod "network-check-target-zgxcv" (UID: "7849d5bd-5ec1-4d67-b508-b361e4f347d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:29:57.589436 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:57.589347    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:29:57.589436 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:57.589405    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs podName:3a72df4f-cfd6-4867-92ee-7f559401f9d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:29:59.589388473 +0000 UTC m=+6.156149009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs") pod "network-metrics-daemon-q2z6p" (UID: "3a72df4f-cfd6-4867-92ee-7f559401f9d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:29:57.909627 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:57.909498    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 11:24:56 +0000 UTC" deadline="2028-03-25 03:11:48.359528754 +0000 UTC"
Sep 08 11:29:57.909627 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:57.909564    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13527h41m50.449969102s"
Sep 08 11:29:57.957578 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:57.957454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:29:57.957730 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:57.957608    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:29:57.993574 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:57.993493    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" event={"ID":"15e2f71567686cf0984f394af0a6f5d0","Type":"ContainerStarted","Data":"90df57f0560dd305ef888804d5cd0b8beadf73034d8f4459ea3d6d8546e64a30"}
Sep 08 11:29:57.996961 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:57.996883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-56s5x" event={"ID":"3cac5fa2-8f4b-4fdf-89c6-353e3e317e01","Type":"ContainerStarted","Data":"cabf2dd37537448ed2ae707d807cf83289ad5155a77f78d6c83061dfb13ad7bc"}
Sep 08 11:29:58.004631 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:58.004567    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66cjb" event={"ID":"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d","Type":"ContainerStarted","Data":"9698d43c2c7e253a9f7bfa701ad43cb32cd381cd419e7a5977c35f0136824f41"}
Sep 08 11:29:58.015995 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:58.015909    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l76h7" event={"ID":"00567f83-0336-467f-826b-a57e063c76ac","Type":"ContainerStarted","Data":"73d681ef0a9ee092b64622c1795aac2693e75c1d01bb3a84678bfc7cbdfe1455"}
Sep 08 11:29:58.023846 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:58.023769    2561 generic.go:356] "Generic (PLEG): container finished" podID="9cb4470e82b2f2f8fdaaff0351211490" containerID="54750d81e0718a96517ed793c2b1532fc01420269c173315301b2d71e80f0f6f" exitCode=0
Sep 08 11:29:58.023983 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:58.023903    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" event={"ID":"9cb4470e82b2f2f8fdaaff0351211490","Type":"ContainerDied","Data":"54750d81e0718a96517ed793c2b1532fc01420269c173315301b2d71e80f0f6f"}
Sep 08 11:29:58.040575 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:58.040516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4w4xp" event={"ID":"50cf80de-8fa6-4300-a09e-4a95c55ef193","Type":"ContainerStarted","Data":"b70dc90923f9a5e9be63674bf82ff3d351e514adb22aaa839526a681f68615a3"}
Sep 08 11:29:58.042894 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:58.042829    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-41.ec2.internal" podStartSLOduration=3.042810033 podStartE2EDuration="3.042810033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:29:58.011445854 +0000 UTC m=+4.578206400" watchObservedRunningTime="2026-09-08 11:29:58.042810033 +0000 UTC m=+4.609570579"
Sep 08 11:29:58.047090 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:58.047047    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ltd4c" event={"ID":"85a2c614-d7f0-4e48-a229-ede402992c01","Type":"ContainerStarted","Data":"c95a4801d72ed68a111f4c31e70b823f88122efff5505f55ada0bc4805c6e8f9"}
Sep 08 11:29:58.065717 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:58.065683    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz" event={"ID":"d0d6b121-1b5d-4967-860a-c30e742876bf","Type":"ContainerStarted","Data":"e10d729fee6fac365b462790f3e36fe40e7ee3fa0dd51a3947b442851af5128e"}
Sep 08 11:29:58.086554 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:58.086503    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc" event={"ID":"02895789-6e33-4cf0-8ebb-d61bbea32d37","Type":"ContainerStarted","Data":"7b06c81662038222fc6cda1ae01ab9c9d981414a23cfc94f5d73f9e3307e15d5"}
Sep 08 11:29:58.103415 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:58.103324    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5nbml" event={"ID":"0ad64a4b-1208-4583-a7cb-4f047fb46203","Type":"ContainerStarted","Data":"9a985490e078fd0841bbc7cf9b49615fd9ad8764d97945f02cf55baedd0417c2"}
Sep 08 11:29:58.116906 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:58.116778    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5xw68" event={"ID":"838a5800-8a91-4b1b-bc39-655796992d82","Type":"ContainerStarted","Data":"0172ac3bc6e54e4f740f22e7d86c63f41c2e4d82316d73a27ea2d2a26554bbc7"}
Sep 08 11:29:58.955144 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:58.955017    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:58.955960 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:58.955174    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:29:58.955960 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:58.955266    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:29:58.955960 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:58.955347    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:29:59.139193 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:59.139152    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" event={"ID":"9cb4470e82b2f2f8fdaaff0351211490","Type":"ContainerStarted","Data":"e201f265d835fc20193bb39de7827430cc02833bc1de09119d336036b04aefe0"}
Sep 08 11:29:59.506699 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:59.505955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:29:59.506699 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:59.506114    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 11:29:59.506699 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:59.506221    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret podName:0dff2c8b-ae64-424e-9b34-28819a39597d nodeName:}" failed. No retries permitted until 2026-09-08 11:30:03.506200143 +0000 UTC m=+10.072960676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret") pod "global-pull-secret-syncer-p9x77" (UID: "0dff2c8b-ae64-424e-9b34-28819a39597d") : object "kube-system"/"original-pull-secret" not registered
Sep 08 11:29:59.608023 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:59.606893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz56d\" (UniqueName: \"kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d\") pod \"network-check-target-zgxcv\" (UID: \"7849d5bd-5ec1-4d67-b508-b361e4f347d0\") " pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:29:59.608023 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:59.606968    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:29:59.608023 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:59.607142    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:29:59.608023 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:59.607214    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs podName:3a72df4f-cfd6-4867-92ee-7f559401f9d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:03.607191814 +0000 UTC m=+10.173952359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs") pod "network-metrics-daemon-q2z6p" (UID: "3a72df4f-cfd6-4867-92ee-7f559401f9d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:29:59.608023 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:59.607861    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 11:29:59.608023 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:59.607889    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 11:29:59.608023 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:59.607904    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qz56d for pod openshift-network-diagnostics/network-check-target-zgxcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:29:59.608023 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:59.607959    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d podName:7849d5bd-5ec1-4d67-b508-b361e4f347d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:03.607940995 +0000 UTC m=+10.174701516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qz56d" (UniqueName: "kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d") pod "network-check-target-zgxcv" (UID: "7849d5bd-5ec1-4d67-b508-b361e4f347d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:29:59.958378 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:29:59.957718    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:29:59.958378 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:29:59.957913    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:00.955461 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:00.955424    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:00.955461 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:00.955435    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:00.955722 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:00.955591    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:00.955784 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:00.955748    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:01.954944 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:01.954909    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:01.955585 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:01.955047    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:02.954690 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:02.954655    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:02.954879 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:02.954776    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:02.955010 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:02.954991    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:02.955356 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:02.955115    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:03.544743 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:03.544703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:03.544916 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:03.544851    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 11:30:03.544957 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:03.544930    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret podName:0dff2c8b-ae64-424e-9b34-28819a39597d nodeName:}" failed. No retries permitted until 2026-09-08 11:30:11.54491063 +0000 UTC m=+18.111671171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret") pod "global-pull-secret-syncer-p9x77" (UID: "0dff2c8b-ae64-424e-9b34-28819a39597d") : object "kube-system"/"original-pull-secret" not registered
Sep 08 11:30:03.645940 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:03.645901    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz56d\" (UniqueName: \"kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d\") pod \"network-check-target-zgxcv\" (UID: \"7849d5bd-5ec1-4d67-b508-b361e4f347d0\") " pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:03.646138 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:03.645964    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:03.646138 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:03.646082    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:30:03.646138 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:03.646094    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 11:30:03.646138 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:03.646122    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 11:30:03.646138 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:03.646134    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qz56d for pod openshift-network-diagnostics/network-check-target-zgxcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:30:03.646359 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:03.646157    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs podName:3a72df4f-cfd6-4867-92ee-7f559401f9d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:11.646134747 +0000 UTC m=+18.212895283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs") pod "network-metrics-daemon-q2z6p" (UID: "3a72df4f-cfd6-4867-92ee-7f559401f9d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:30:03.646359 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:03.646186    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d podName:7849d5bd-5ec1-4d67-b508-b361e4f347d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:11.646171111 +0000 UTC m=+18.212931649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qz56d" (UniqueName: "kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d") pod "network-check-target-zgxcv" (UID: "7849d5bd-5ec1-4d67-b508-b361e4f347d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:30:03.956610 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:03.956528    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:03.957042 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:03.956689    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:04.954968 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:04.954930    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:04.955160 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:04.955071    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:04.955912 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:04.955422    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:04.955912 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:04.955534    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:05.956775 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:05.956207    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:05.956775 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:05.956351    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:06.954826 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:06.954625    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:06.955011 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:06.954669    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:06.955011 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:06.954928    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:06.955011 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:06.955000    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:07.955499 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:07.955393    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:07.955955 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:07.955549    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:08.955262 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:08.955228    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:08.955442 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:08.955363    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:08.955442 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:08.955415    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:08.955545 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:08.955529    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:09.955503 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:09.955453    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:09.955707 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:09.955605    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:10.954677 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:10.954638    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:10.954918 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:10.954756    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:10.954918 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:10.954810    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:10.954983 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:10.954913    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:11.603070 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:11.603029    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:11.603488 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:11.603176    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 11:30:11.603488 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:11.603258    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret podName:0dff2c8b-ae64-424e-9b34-28819a39597d nodeName:}" failed. No retries permitted until 2026-09-08 11:30:27.603235523 +0000 UTC m=+34.169996065 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret") pod "global-pull-secret-syncer-p9x77" (UID: "0dff2c8b-ae64-424e-9b34-28819a39597d") : object "kube-system"/"original-pull-secret" not registered
Sep 08 11:30:11.703565 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:11.703530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:11.703730 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:11.703686    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:30:11.703730 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:11.703698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz56d\" (UniqueName: \"kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d\") pod \"network-check-target-zgxcv\" (UID: \"7849d5bd-5ec1-4d67-b508-b361e4f347d0\") " pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:11.703863 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:11.703755    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs podName:3a72df4f-cfd6-4867-92ee-7f559401f9d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:27.703734226 +0000 UTC m=+34.270494747 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs") pod "network-metrics-daemon-q2z6p" (UID: "3a72df4f-cfd6-4867-92ee-7f559401f9d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:30:11.703863 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:11.703834    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 11:30:11.703863 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:11.703856    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 11:30:11.703863 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:11.703866    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qz56d for pod openshift-network-diagnostics/network-check-target-zgxcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:30:11.704045 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:11.703911    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d podName:7849d5bd-5ec1-4d67-b508-b361e4f347d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:27.703897743 +0000 UTC m=+34.270658263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qz56d" (UniqueName: "kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d") pod "network-check-target-zgxcv" (UID: "7849d5bd-5ec1-4d67-b508-b361e4f347d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:30:11.955526 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:11.955427    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:11.955670 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:11.955575    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:12.954676 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:12.954638    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:12.954676 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:12.954681    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:12.955166 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:12.954781    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:12.955166 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:12.954891    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:13.956385 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:13.956352    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:13.956866 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:13.956479    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:14.955072 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:14.955039    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:14.955261 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:14.955043    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:14.955261 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:14.955148    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:14.955400 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:14.955287    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:15.955767 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:15.955582    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:15.956391 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:15.955837    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:16.189782 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.189505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4w4xp" event={"ID":"50cf80de-8fa6-4300-a09e-4a95c55ef193","Type":"ContainerStarted","Data":"e7b8bd26b5641ed1aab8dab5d436dc461ac7fab26d68ef14018c9ed10f78295a"}
Sep 08 11:30:16.191287 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.191257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ltd4c" event={"ID":"85a2c614-d7f0-4e48-a229-ede402992c01","Type":"ContainerStarted","Data":"2e6032ee5313591ddbf00bfd56d0c34d1dec560a77a9b88d5164969863625f8b"}
Sep 08 11:30:16.194795 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.194759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz" event={"ID":"d0d6b121-1b5d-4967-860a-c30e742876bf","Type":"ContainerStarted","Data":"49a70d0b938cf432fe94b1c3742fdb3ef84fafadfb851d76a8e70d3b40fcf8a2"}
Sep 08 11:30:16.194906 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.194836    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz" event={"ID":"d0d6b121-1b5d-4967-860a-c30e742876bf","Type":"ContainerStarted","Data":"ac918c4f9f37ecd761dde30f1c016b984ec4efb24226053e6c4d5b53fdf5e403"}
Sep 08 11:30:16.194906 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.194869    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz" event={"ID":"d0d6b121-1b5d-4967-860a-c30e742876bf","Type":"ContainerStarted","Data":"3c143438eedae165da1a366020defe6363fba0e693dde9b3d15d68b38be4573b"}
Sep 08 11:30:16.194906 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.194898    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz" event={"ID":"d0d6b121-1b5d-4967-860a-c30e742876bf","Type":"ContainerStarted","Data":"f945ced68e077bf8d642edb7db9c614e1c4c759cd302c5389afac30d79bde4c0"}
Sep 08 11:30:16.195050 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.194927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz" event={"ID":"d0d6b121-1b5d-4967-860a-c30e742876bf","Type":"ContainerStarted","Data":"02edcca40cfe52fa7a515ff442aa26969fab68fa3e01bc3de271ec3cc0596a5f"}
Sep 08 11:30:16.195050 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.194967    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz" event={"ID":"d0d6b121-1b5d-4967-860a-c30e742876bf","Type":"ContainerStarted","Data":"765950511bb940d37eb7c592df575138139b44a2c0acd6107d7af2225c3cb2f6"}
Sep 08 11:30:16.196505 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.196476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc" event={"ID":"02895789-6e33-4cf0-8ebb-d61bbea32d37","Type":"ContainerStarted","Data":"a0a3e51755e2bb6fd23eaa459862fb67e56f27fd6d409b2a695d1eb71583c175"}
Sep 08 11:30:16.198114 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.198091    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5nbml" event={"ID":"0ad64a4b-1208-4583-a7cb-4f047fb46203","Type":"ContainerStarted","Data":"dca30a00eee6489585eb99c3f79d043bea47976d2e233faf37ef5328d73a3f00"}
Sep 08 11:30:16.199664 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.199639    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5xw68" event={"ID":"838a5800-8a91-4b1b-bc39-655796992d82","Type":"ContainerStarted","Data":"6eaf4d4b33b2309ac8492e3f073cdbe57b15439e528181637785a3c4f39b9be7"}
Sep 08 11:30:16.201407 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.201384    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d" containerID="3950d0082e274ff891c868433e5bc5a3ecf47f26d2b2774fd491d24573b8f0c0" exitCode=0
Sep 08 11:30:16.201508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.201477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66cjb" event={"ID":"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d","Type":"ContainerDied","Data":"3950d0082e274ff891c868433e5bc5a3ecf47f26d2b2774fd491d24573b8f0c0"}
Sep 08 11:30:16.203229 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.203211    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l76h7" event={"ID":"00567f83-0336-467f-826b-a57e063c76ac","Type":"ContainerStarted","Data":"dc4911397093e5ab733b8759c8dc0c3b08168dde618a90f30d2bb764292aefc3"}
Sep 08 11:30:16.212830 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.212783    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-41.ec2.internal" podStartSLOduration=21.212769879 podStartE2EDuration="21.212769879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:29:59.157719294 +0000 UTC m=+5.724479839" watchObservedRunningTime="2026-09-08 11:30:16.212769879 +0000 UTC m=+22.779530419"
Sep 08 11:30:16.230776 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.230724    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l76h7" podStartSLOduration=4.292819728 podStartE2EDuration="22.230706982s" totalImagesPullingTime="17.937887254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:16.230479867 +0000 UTC m=+22.797240401" watchObservedRunningTime="2026-09-08 11:30:16.230706982 +0000 UTC m=+22.797467523"
Sep 08 11:30:16.231161 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.231129    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4w4xp" podStartSLOduration=4.310868984 podStartE2EDuration="22.231113648s" totalImagesPullingTime="17.920244664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:16.21259331 +0000 UTC m=+22.779353852" watchObservedRunningTime="2026-09-08 11:30:16.231113648 +0000 UTC m=+22.797874193"
Sep 08 11:30:16.269529 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.269447    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5xw68" podStartSLOduration=4.325567415 podStartE2EDuration="22.269426656s" totalImagesPullingTime="17.943859241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:16.247052813 +0000 UTC m=+22.813813354" watchObservedRunningTime="2026-09-08 11:30:16.269426656 +0000 UTC m=+22.836187197"
Sep 08 11:30:16.269728 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.269689    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ltd4c" podStartSLOduration=4.282057608 podStartE2EDuration="22.269677815s" totalImagesPullingTime="17.987620207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:16.269571699 +0000 UTC m=+22.836332222" watchObservedRunningTime="2026-09-08 11:30:16.269677815 +0000 UTC m=+22.836438358"
Sep 08 11:30:16.313634 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.313496    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5nbml" podStartSLOduration=4.360633307 podStartE2EDuration="22.313480745s" totalImagesPullingTime="17.952847438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:16.313441251 +0000 UTC m=+22.880201792" watchObservedRunningTime="2026-09-08 11:30:16.313480745 +0000 UTC m=+22.880241281"
Sep 08 11:30:16.955683 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.955655    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:16.955857 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:16.955690    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:16.955857 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:16.955785    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:16.956255 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:16.955900    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:17.080016 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:17.079802    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 11:30:17.208031 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:17.207910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc" event={"ID":"02895789-6e33-4cf0-8ebb-d61bbea32d37","Type":"ContainerStarted","Data":"990489bfa1281709c4c7ba8b589963a0115ed399f2c6faa5f020d3e52ec8f668"}
Sep 08 11:30:17.209496 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:17.209448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-56s5x" event={"ID":"3cac5fa2-8f4b-4fdf-89c6-353e3e317e01","Type":"ContainerStarted","Data":"8b257a764d37e3ff2bb0b001d3633eb04bf685ddd3180298a25e790c93349625"}
Sep 08 11:30:17.925604 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:17.925509    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T11:30:17.080008768Z","UUID":"1acb0ca2-0b72-415c-8148-5b9e9ce337c1","Handler":null,"Name":"","Endpoint":""}
Sep 08 11:30:17.927244 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:17.927218    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 11:30:17.929049 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:17.927256    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 11:30:17.954767 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:17.954741    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:17.954987 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:17.954870    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:18.216054 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:18.215966    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz" event={"ID":"d0d6b121-1b5d-4967-860a-c30e742876bf","Type":"ContainerStarted","Data":"20af13f8c6d1ec0b3571a30a682728e691076ba0e6df3767dd8cb00a11eeb53b"}
Sep 08 11:30:18.955424 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:18.955340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:18.955424 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:18.955374    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:18.955664 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:18.955493    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:18.955664 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:18.955611    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:19.220165 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:19.220079    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc" event={"ID":"02895789-6e33-4cf0-8ebb-d61bbea32d37","Type":"ContainerStarted","Data":"dd64fbc472fc3217d6236331082c27840275ce24b5291a8bee20b004f4e174cb"}
Sep 08 11:30:19.241677 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:19.241621    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-56s5x" podStartSLOduration=7.316247998 podStartE2EDuration="25.241604584s" totalImagesPullingTime="17.925356586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:17.228543595 +0000 UTC m=+23.795304136" watchObservedRunningTime="2026-09-08 11:30:19.241604584 +0000 UTC m=+25.808365124"
Sep 08 11:30:19.241866 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:19.241778    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4klc" podStartSLOduration=4.088895377 podStartE2EDuration="25.241771046s" totalImagesPullingTime="21.152875669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:29:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:19.240370485 +0000 UTC m=+25.807131047" watchObservedRunningTime="2026-09-08 11:30:19.241771046 +0000 UTC m=+25.808531588"
Sep 08 11:30:19.405408 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:19.405367    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4w4xp"
Sep 08 11:30:19.406084 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:19.406062    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4w4xp"
Sep 08 11:30:19.955310 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:19.955271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:19.955515 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:19.955413    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:20.222914 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:20.222827    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4w4xp"
Sep 08 11:30:20.223410 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:20.223387    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4w4xp"
Sep 08 11:30:20.954956 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:20.954744    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:20.955090 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:20.954762    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:20.955090 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:20.954991    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:20.955090 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:20.955052    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:21.226682 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:21.226605    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d" containerID="b449f0f1b97ad96083f80cb767c5504059cbe3381c22a450f57aed5bdc76d254" exitCode=0
Sep 08 11:30:21.227446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:21.226689    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66cjb" event={"ID":"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d","Type":"ContainerDied","Data":"b449f0f1b97ad96083f80cb767c5504059cbe3381c22a450f57aed5bdc76d254"}
Sep 08 11:30:21.230514 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:21.230491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz" event={"ID":"d0d6b121-1b5d-4967-860a-c30e742876bf","Type":"ContainerStarted","Data":"293ed7d170f6e9e4b4fc2971a938634fd6addc139bc9cd424def447f509c66bf"}
Sep 08 11:30:21.230745 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:21.230717    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:30:21.230826 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:21.230790    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:30:21.245749 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:21.245726    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:30:21.318698 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:21.318639    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz" podStartSLOduration=9.327184898 podStartE2EDuration="27.318624106s" totalImagesPullingTime="17.991439208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:21.317150603 +0000 UTC m=+27.883911144" watchObservedRunningTime="2026-09-08 11:30:21.318624106 +0000 UTC m=+27.885384648"
Sep 08 11:30:21.955051 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:21.955016    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:21.955204 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:21.955144    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:22.232811 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:22.232731    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:30:22.247506 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:22.247482    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:30:22.412262 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:22.412235    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p9x77"]
Sep 08 11:30:22.412413 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:22.412356    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:22.412521 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:22.412459    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:22.414868 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:22.414843    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zgxcv"]
Sep 08 11:30:22.414977 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:22.414933    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:22.415047 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:22.415027    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:22.417667 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:22.417646    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q2z6p"]
Sep 08 11:30:22.417749 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:22.417739    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:22.417836 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:22.417821    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:23.955944 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:23.955909    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:23.956381 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:23.955994    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:23.956381 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:23.956025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:23.956381 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:23.956067    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:23.956381 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:23.956101    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:23.956381 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:23.956192    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:24.238748 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:24.238410    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d" containerID="147a3dbb1e96eb543b99cf2bef005dd4e3f72d3947560715ae4fcefa8bb04d6f" exitCode=0
Sep 08 11:30:24.238920 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:24.238491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66cjb" event={"ID":"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d","Type":"ContainerDied","Data":"147a3dbb1e96eb543b99cf2bef005dd4e3f72d3947560715ae4fcefa8bb04d6f"}
Sep 08 11:30:25.243297 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:25.243204    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d" containerID="9c5c4af4071d0b9693af5534905cd07a84f1a8a0acfabeb4fe10b989165b3784" exitCode=0
Sep 08 11:30:25.243297 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:25.243247    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66cjb" event={"ID":"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d","Type":"ContainerDied","Data":"9c5c4af4071d0b9693af5534905cd07a84f1a8a0acfabeb4fe10b989165b3784"}
Sep 08 11:30:25.954889 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:25.954857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:25.955061 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:25.954857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:25.955061 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:25.954995    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:25.955061 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:25.954990    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p9x77" podUID="0dff2c8b-ae64-424e-9b34-28819a39597d"
Sep 08 11:30:25.955225 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:25.955092    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zgxcv" podUID="7849d5bd-5ec1-4d67-b508-b361e4f347d0"
Sep 08 11:30:25.955225 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:25.955202    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2z6p" podUID="3a72df4f-cfd6-4867-92ee-7f559401f9d0"
Sep 08 11:30:27.620772 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.620737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:27.621099 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:27.620880    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 11:30:27.621099 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:27.620938    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret podName:0dff2c8b-ae64-424e-9b34-28819a39597d nodeName:}" failed. No retries permitted until 2026-09-08 11:30:59.62092415 +0000 UTC m=+66.187684682 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret") pod "global-pull-secret-syncer-p9x77" (UID: "0dff2c8b-ae64-424e-9b34-28819a39597d") : object "kube-system"/"original-pull-secret" not registered
Sep 08 11:30:27.721787 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.721745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:27.721972 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.721906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz56d\" (UniqueName: \"kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d\") pod \"network-check-target-zgxcv\" (UID: \"7849d5bd-5ec1-4d67-b508-b361e4f347d0\") " pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:27.721972 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:27.721929    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:30:27.722064 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:27.722012    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs podName:3a72df4f-cfd6-4867-92ee-7f559401f9d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:59.721991619 +0000 UTC m=+66.288752158 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs") pod "network-metrics-daemon-q2z6p" (UID: "3a72df4f-cfd6-4867-92ee-7f559401f9d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 11:30:27.722064 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:27.722035    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 11:30:27.722064 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:27.722054    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 11:30:27.722181 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:27.722067    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qz56d for pod openshift-network-diagnostics/network-check-target-zgxcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:30:27.722181 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:27.722116    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d podName:7849d5bd-5ec1-4d67-b508-b361e4f347d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:59.722105181 +0000 UTC m=+66.288865701 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qz56d" (UniqueName: "kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d") pod "network-check-target-zgxcv" (UID: "7849d5bd-5ec1-4d67-b508-b361e4f347d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 11:30:27.751567 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.751526    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-41.ec2.internal" event="NodeReady"
Sep 08 11:30:27.751743 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.751682    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 11:30:27.791740 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.791705    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c88c548cf-26prd"]
Sep 08 11:30:27.827762 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.827674    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-67xmf"]
Sep 08 11:30:27.827939 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.827827    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:27.830876 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.830853    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-59gmm\""
Sep 08 11:30:27.830876 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.830864    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 11:30:27.831072 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.830850    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 11:30:27.831072 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.830867    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 11:30:27.837928 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.837749    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 11:30:27.841645 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.841622    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c88c548cf-26prd"]
Sep 08 11:30:27.841645 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.841657    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b5kmv"]
Sep 08 11:30:27.841824 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.841764    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:27.844997 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.844977    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2v99l\""
Sep 08 11:30:27.845431 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.845404    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 11:30:27.845706 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.845684    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 11:30:27.863740 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.863712    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-67xmf"]
Sep 08 11:30:27.863885 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.863753    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b5kmv"]
Sep 08 11:30:27.863885 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.863842    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b5kmv"
Sep 08 11:30:27.866443 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.866412    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 11:30:27.866757 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.866734    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 11:30:27.866870 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.866798    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h4mlz\""
Sep 08 11:30:27.867047 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.867030    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 11:30:27.923766 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.923724    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-image-registry-private-configuration\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:27.923766 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.923770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-installation-pull-secrets\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:27.924022 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.923800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:27.924022 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.923851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-certificates\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:27.924022 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.923921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b795d\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-kube-api-access-b795d\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:27.924152 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.924024    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-ca-trust-extracted\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:27.924152 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.924104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-bound-sa-token\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:27.924152 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.924145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-trusted-ca\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:27.955220 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.955184    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:27.955383 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.955184    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:27.955383 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.955381    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:27.958384 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.958361    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nwz6n\""
Sep 08 11:30:27.958731 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.958710    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 11:30:27.958819 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.958742    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 11:30:27.958819 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.958755    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 11:30:27.958896 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.958718    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 11:30:27.958896 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:27.958711    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt4h2\""
Sep 08 11:30:28.024598 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.024552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-bound-sa-token\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.024598 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.024597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-trusted-ca\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.024846 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.024618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c229e7b3-1c76-407e-b4cf-6208554a0009-cert\") pod \"ingress-canary-b5kmv\" (UID: \"c229e7b3-1c76-407e-b4cf-6208554a0009\") " pod="openshift-ingress-canary/ingress-canary-b5kmv"
Sep 08 11:30:28.024846 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.024762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvczk\" (UniqueName: \"kubernetes.io/projected/1a77e471-aaa5-4f4d-932d-c8466b009a67-kube-api-access-jvczk\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:28.024846 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.024810    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a77e471-aaa5-4f4d-932d-c8466b009a67-config-volume\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:28.025007 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.024860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-image-registry-private-configuration\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.025007 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.024886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a77e471-aaa5-4f4d-932d-c8466b009a67-tmp-dir\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:28.025007 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.024917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-installation-pull-secrets\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.025007 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.024943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.025007 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.024962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:28.025007 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.025002    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-certificates\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.025290 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.025030    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b795d\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-kube-api-access-b795d\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.025290 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.025053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r945v\" (UniqueName: \"kubernetes.io/projected/c229e7b3-1c76-407e-b4cf-6208554a0009-kube-api-access-r945v\") pod \"ingress-canary-b5kmv\" (UID: \"c229e7b3-1c76-407e-b4cf-6208554a0009\") " pod="openshift-ingress-canary/ingress-canary-b5kmv"
Sep 08 11:30:28.025290 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:28.025079    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 11:30:28.025290 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:28.025100    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c88c548cf-26prd: secret "image-registry-tls" not found
Sep 08 11:30:28.025290 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:28.025157    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls podName:1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d nodeName:}" failed. No retries permitted until 2026-09-08 11:30:28.525135954 +0000 UTC m=+35.091896501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls") pod "image-registry-6c88c548cf-26prd" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d") : secret "image-registry-tls" not found
Sep 08 11:30:28.025587 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.025097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-ca-trust-extracted\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.025587 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.025396    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-ca-trust-extracted\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.025690 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.025603    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-certificates\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.025744 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.025687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-trusted-ca\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.029545 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.029511    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-installation-pull-secrets\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.029669 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.029544    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-image-registry-private-configuration\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.034866 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.034838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-bound-sa-token\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.035135 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.035112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b795d\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-kube-api-access-b795d\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.126255 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.126163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a77e471-aaa5-4f4d-932d-c8466b009a67-tmp-dir\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:28.126255 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.126220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:28.126493 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:28.126322    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 11:30:28.126493 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.126322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r945v\" (UniqueName: \"kubernetes.io/projected/c229e7b3-1c76-407e-b4cf-6208554a0009-kube-api-access-r945v\") pod \"ingress-canary-b5kmv\" (UID: \"c229e7b3-1c76-407e-b4cf-6208554a0009\") " pod="openshift-ingress-canary/ingress-canary-b5kmv"
Sep 08 11:30:28.126493 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:28.126413    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls podName:1a77e471-aaa5-4f4d-932d-c8466b009a67 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:28.626393568 +0000 UTC m=+35.193154090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls") pod "dns-default-67xmf" (UID: "1a77e471-aaa5-4f4d-932d-c8466b009a67") : secret "dns-default-metrics-tls" not found
Sep 08 11:30:28.126493 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.126452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c229e7b3-1c76-407e-b4cf-6208554a0009-cert\") pod \"ingress-canary-b5kmv\" (UID: \"c229e7b3-1c76-407e-b4cf-6208554a0009\") " pod="openshift-ingress-canary/ingress-canary-b5kmv"
Sep 08 11:30:28.126686 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.126499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvczk\" (UniqueName: \"kubernetes.io/projected/1a77e471-aaa5-4f4d-932d-c8466b009a67-kube-api-access-jvczk\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:28.126686 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.126533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a77e471-aaa5-4f4d-932d-c8466b009a67-config-volume\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:28.126782 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.126745    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a77e471-aaa5-4f4d-932d-c8466b009a67-tmp-dir\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:28.127003 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.126985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a77e471-aaa5-4f4d-932d-c8466b009a67-config-volume\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:28.129196 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.129176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c229e7b3-1c76-407e-b4cf-6208554a0009-cert\") pod \"ingress-canary-b5kmv\" (UID: \"c229e7b3-1c76-407e-b4cf-6208554a0009\") " pod="openshift-ingress-canary/ingress-canary-b5kmv"
Sep 08 11:30:28.140131 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.140101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r945v\" (UniqueName: \"kubernetes.io/projected/c229e7b3-1c76-407e-b4cf-6208554a0009-kube-api-access-r945v\") pod \"ingress-canary-b5kmv\" (UID: \"c229e7b3-1c76-407e-b4cf-6208554a0009\") " pod="openshift-ingress-canary/ingress-canary-b5kmv"
Sep 08 11:30:28.140459 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.140430    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvczk\" (UniqueName: \"kubernetes.io/projected/1a77e471-aaa5-4f4d-932d-c8466b009a67-kube-api-access-jvczk\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:28.174479 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.174434    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b5kmv"
Sep 08 11:30:28.530077 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.530035    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:28.530263 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:28.530195    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 11:30:28.530263 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:28.530213    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c88c548cf-26prd: secret "image-registry-tls" not found
Sep 08 11:30:28.530341 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:28.530276    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls podName:1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d nodeName:}" failed. No retries permitted until 2026-09-08 11:30:29.530255964 +0000 UTC m=+36.097016497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls") pod "image-registry-6c88c548cf-26prd" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d") : secret "image-registry-tls" not found
Sep 08 11:30:28.630546 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:28.630497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:28.631017 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:28.630667    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 11:30:28.631017 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:28.630749    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls podName:1a77e471-aaa5-4f4d-932d-c8466b009a67 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:29.630730863 +0000 UTC m=+36.197491388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls") pod "dns-default-67xmf" (UID: "1a77e471-aaa5-4f4d-932d-c8466b009a67") : secret "dns-default-metrics-tls" not found
Sep 08 11:30:29.322708 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:29.322536    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:29.426773 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:29.426740    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:29.500736 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:29.500625    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b5kmv"]
Sep 08 11:30:29.504950 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:30:29.504926    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc229e7b3_1c76_407e_b4cf_6208554a0009.slice/crio-9c47160b096ee2fe207c82ea9632b033a6ee75620889df48ddde0676ae2b1df1 WatchSource:0}: Error finding container 9c47160b096ee2fe207c82ea9632b033a6ee75620889df48ddde0676ae2b1df1: Status 404 returned error can't find the container with id 9c47160b096ee2fe207c82ea9632b033a6ee75620889df48ddde0676ae2b1df1
Sep 08 11:30:29.538283 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:29.538261    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:29.538444 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:29.538426    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 11:30:29.538526 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:29.538447    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c88c548cf-26prd: secret "image-registry-tls" not found
Sep 08 11:30:29.538526 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:29.538523    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls podName:1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d nodeName:}" failed. No retries permitted until 2026-09-08 11:30:31.53850382 +0000 UTC m=+38.105264357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls") pod "image-registry-6c88c548cf-26prd" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d") : secret "image-registry-tls" not found
Sep 08 11:30:29.639679 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:29.639632    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:29.640238 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:29.639774    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 11:30:29.640238 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:29.639833    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls podName:1a77e471-aaa5-4f4d-932d-c8466b009a67 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:31.639816119 +0000 UTC m=+38.206576648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls") pod "dns-default-67xmf" (UID: "1a77e471-aaa5-4f4d-932d-c8466b009a67") : secret "dns-default-metrics-tls" not found
Sep 08 11:30:29.793767 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:29.793686    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:29.865964 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:29.865933    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:30.257289 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:30.257253    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b5kmv" event={"ID":"c229e7b3-1c76-407e-b4cf-6208554a0009","Type":"ContainerStarted","Data":"9c47160b096ee2fe207c82ea9632b033a6ee75620889df48ddde0676ae2b1df1"}
Sep 08 11:30:30.260248 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:30.260211    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d" containerID="b81868bc83b70280fd77d97bcc47518e169fa0ef3b4188e493178e7961332b7b" exitCode=0
Sep 08 11:30:30.260376 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:30.260267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66cjb" event={"ID":"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d","Type":"ContainerDied","Data":"b81868bc83b70280fd77d97bcc47518e169fa0ef3b4188e493178e7961332b7b"}
Sep 08 11:30:31.266672 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:31.266632    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d" containerID="2785f5dd9bfee0fa188f99b867d62bb3156214567f8133104d1633ea5ec32e57" exitCode=0
Sep 08 11:30:31.267219 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:31.266710    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66cjb" event={"ID":"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d","Type":"ContainerDied","Data":"2785f5dd9bfee0fa188f99b867d62bb3156214567f8133104d1633ea5ec32e57"}
Sep 08 11:30:31.555491 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:31.555385    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:31.555672 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:31.555598    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 11:30:31.555672 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:31.555621    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c88c548cf-26prd: secret "image-registry-tls" not found
Sep 08 11:30:31.555774 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:31.555708    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls podName:1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d nodeName:}" failed. No retries permitted until 2026-09-08 11:30:35.555687678 +0000 UTC m=+42.122448198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls") pod "image-registry-6c88c548cf-26prd" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d") : secret "image-registry-tls" not found
Sep 08 11:30:31.656520 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:31.656485    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:31.656697 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:31.656672    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 11:30:31.656815 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:31.656765    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls podName:1a77e471-aaa5-4f4d-932d-c8466b009a67 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:35.656740899 +0000 UTC m=+42.223501440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls") pod "dns-default-67xmf" (UID: "1a77e471-aaa5-4f4d-932d-c8466b009a67") : secret "dns-default-metrics-tls" not found
Sep 08 11:30:32.272219 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:32.272192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-66cjb" event={"ID":"a8699c6b-4ab0-4b96-b3cf-3bbd3e7c744d","Type":"ContainerStarted","Data":"5a24607c955930e26a13014e43eb51494051d7488f4919a31a691a7d19e9dd96"}
Sep 08 11:30:32.302907 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:32.302736    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-66cjb" podStartSLOduration=8.354220339 podStartE2EDuration="38.302719224s" totalImagesPullingTime="29.948498885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:29:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:32.301954114 +0000 UTC m=+38.868714656" watchObservedRunningTime="2026-09-08 11:30:32.302719224 +0000 UTC m=+38.869479761"
Sep 08 11:30:32.429222 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:32.429190    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:32.516641 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:32.516522    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:33.276294 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:33.276248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b5kmv" event={"ID":"c229e7b3-1c76-407e-b4cf-6208554a0009","Type":"ContainerStarted","Data":"d6911cd2579c69651e57338ca680e8aa4ecf24ae5f46a07abe704b46afba0c54"}
Sep 08 11:30:33.286985 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:33.286954    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:33.294165 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:33.294122    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b5kmv" podStartSLOduration=3.54697241 podStartE2EDuration="6.294106925s" totalImagesPullingTime="2.747134515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:33.29334583 +0000 UTC m=+39.860106394" watchObservedRunningTime="2026-09-08 11:30:33.294106925 +0000 UTC m=+39.860867458"
Sep 08 11:30:34.717361 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:34.717327    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:34.790601 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:34.790573    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:35.586302 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:35.586267    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:35.586486 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:35.586424    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 11:30:35.586486 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:35.586446    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c88c548cf-26prd: secret "image-registry-tls" not found
Sep 08 11:30:35.586550 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:35.586516    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls podName:1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d nodeName:}" failed. No retries permitted until 2026-09-08 11:30:43.586500975 +0000 UTC m=+50.153261522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls") pod "image-registry-6c88c548cf-26prd" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d") : secret "image-registry-tls" not found
Sep 08 11:30:35.687460 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:35.687421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:35.687683 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:35.687595    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 11:30:35.687683 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:35.687675    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls podName:1a77e471-aaa5-4f4d-932d-c8466b009a67 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:43.687654891 +0000 UTC m=+50.254415429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls") pod "dns-default-67xmf" (UID: "1a77e471-aaa5-4f4d-932d-c8466b009a67") : secret "dns-default-metrics-tls" not found
Sep 08 11:30:42.005804 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:42.005774    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:43.647055 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:43.647016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:43.647487 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:43.647178    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 11:30:43.647487 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:43.647195    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c88c548cf-26prd: secret "image-registry-tls" not found
Sep 08 11:30:43.647487 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:43.647252    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls podName:1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d nodeName:}" failed. No retries permitted until 2026-09-08 11:30:59.647236814 +0000 UTC m=+66.213997337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls") pod "image-registry-6c88c548cf-26prd" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d") : secret "image-registry-tls" not found
Sep 08 11:30:43.747713 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:43.747679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:43.747834 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:43.747789    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 11:30:43.747876 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:43.747841    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls podName:1a77e471-aaa5-4f4d-932d-c8466b009a67 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:59.747827729 +0000 UTC m=+66.314588247 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls") pod "dns-default-67xmf" (UID: "1a77e471-aaa5-4f4d-932d-c8466b009a67") : secret "dns-default-metrics-tls" not found
Sep 08 11:30:47.069462 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:47.069429    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:47.171097 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:47.171069    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:48.070154 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:48.070124    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:48.169527 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:48.169489    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:48.251048 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:48.251019    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:48.981270 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:48.981239    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:49.053944 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:49.053913    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:49.700000 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:49.699967    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:49.800519 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:49.800487    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:52.046090 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:52.046045    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:54.249413 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:54.249389    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4wfz"
Sep 08 11:30:54.444920 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:54.444886    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:54.538488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:54.538401    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:59.069967 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.069930    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:59.193019 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.192991    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:59.670185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.670146    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:30:59.670383 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.670306    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:59.670383 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:59.670346    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 11:30:59.670383 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:59.670365    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c88c548cf-26prd: secret "image-registry-tls" not found
Sep 08 11:30:59.670583 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:59.670434    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls podName:1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d nodeName:}" failed. No retries permitted until 2026-09-08 11:31:31.670417863 +0000 UTC m=+98.237178381 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls") pod "image-registry-6c88c548cf-26prd" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d") : secret "image-registry-tls" not found
Sep 08 11:30:59.673019 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.673001    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 11:30:59.684222 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.684195    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0dff2c8b-ae64-424e-9b34-28819a39597d-original-pull-secret\") pod \"global-pull-secret-syncer-p9x77\" (UID: \"0dff2c8b-ae64-424e-9b34-28819a39597d\") " pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:59.770704 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.770665    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:30:59.770704 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.770719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz56d\" (UniqueName: \"kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d\") pod \"network-check-target-zgxcv\" (UID: \"7849d5bd-5ec1-4d67-b508-b361e4f347d0\") " pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:59.770919 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.770745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:30:59.770919 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:59.770834    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 11:30:59.770919 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:59.770912    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls podName:1a77e471-aaa5-4f4d-932d-c8466b009a67 nodeName:}" failed. No retries permitted until 2026-09-08 11:31:31.770896553 +0000 UTC m=+98.337657085 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls") pod "dns-default-67xmf" (UID: "1a77e471-aaa5-4f4d-932d-c8466b009a67") : secret "dns-default-metrics-tls" not found
Sep 08 11:30:59.773344 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.773327    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 11:30:59.773675 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.773660    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 11:30:59.773797 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.773783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p9x77"
Sep 08 11:30:59.781299 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:59.781279    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 11:30:59.781383 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:30:59.781350    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs podName:3a72df4f-cfd6-4867-92ee-7f559401f9d0 nodeName:}" failed. No retries permitted until 2026-09-08 11:32:03.781335095 +0000 UTC m=+130.348095627 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs") pod "network-metrics-daemon-q2z6p" (UID: "3a72df4f-cfd6-4867-92ee-7f559401f9d0") : secret "metrics-daemon-secret" not found
Sep 08 11:30:59.783669 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.783652    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 11:30:59.793805 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.793785    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz56d\" (UniqueName: \"kubernetes.io/projected/7849d5bd-5ec1-4d67-b508-b361e4f347d0-kube-api-access-qz56d\") pod \"network-check-target-zgxcv\" (UID: \"7849d5bd-5ec1-4d67-b508-b361e4f347d0\") " pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:30:59.851783 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.851742    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:30:59.949537 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:30:59.949429    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:00.071669 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:00.071631    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p9x77"]
Sep 08 11:31:00.075640 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:31:00.075612    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dff2c8b_ae64_424e_9b34_28819a39597d.slice/crio-5acf3d011b5e9d47c93a689674cdd9c1df08e6fcb931010b39ad783508f54749 WatchSource:0}: Error finding container 5acf3d011b5e9d47c93a689674cdd9c1df08e6fcb931010b39ad783508f54749: Status 404 returned error can't find the container with id 5acf3d011b5e9d47c93a689674cdd9c1df08e6fcb931010b39ad783508f54749
Sep 08 11:31:00.083849 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:00.083830    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nwz6n\""
Sep 08 11:31:00.091994 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:00.091978    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:31:00.341997 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:00.341959    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p9x77" event={"ID":"0dff2c8b-ae64-424e-9b34-28819a39597d","Type":"ContainerStarted","Data":"5acf3d011b5e9d47c93a689674cdd9c1df08e6fcb931010b39ad783508f54749"}
Sep 08 11:31:00.391203 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:00.391172    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zgxcv"]
Sep 08 11:31:00.394666 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:31:00.394644    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7849d5bd_5ec1_4d67_b508_b361e4f347d0.slice/crio-a63c4bec483f37b6b5a5ddc79171116c63553d8bdd723f8209a2854f0afa5966 WatchSource:0}: Error finding container a63c4bec483f37b6b5a5ddc79171116c63553d8bdd723f8209a2854f0afa5966: Status 404 returned error can't find the container with id a63c4bec483f37b6b5a5ddc79171116c63553d8bdd723f8209a2854f0afa5966
Sep 08 11:31:01.346360 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:01.346323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zgxcv" event={"ID":"7849d5bd-5ec1-4d67-b508-b361e4f347d0","Type":"ContainerStarted","Data":"a63c4bec483f37b6b5a5ddc79171116c63553d8bdd723f8209a2854f0afa5966"}
Sep 08 11:31:04.697481 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:04.697428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:04.830865 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:04.830820    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:05.358669 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:05.358638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p9x77" event={"ID":"0dff2c8b-ae64-424e-9b34-28819a39597d","Type":"ContainerStarted","Data":"43f8ae174b68ad1b4f4c0ef6368b7f366ac2dee902ac8b88a528d5f2bb0d1218"}
Sep 08 11:31:05.360111 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:05.360085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zgxcv" event={"ID":"7849d5bd-5ec1-4d67-b508-b361e4f347d0","Type":"ContainerStarted","Data":"3906f11032a2029ce79500e20c45931d2135a8be8af1be736cb8dcea8af9c370"}
Sep 08 11:31:05.360223 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:05.360148    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:31:05.377884 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:05.377830    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p9x77" podStartSLOduration=66.499680051 podStartE2EDuration="1m11.377816547s" totalImagesPullingTime="4.878136496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:31:05.377496814 +0000 UTC m=+71.944257350" watchObservedRunningTime="2026-09-08 11:31:05.377816547 +0000 UTC m=+71.944577087"
Sep 08 11:31:05.397657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:05.397611    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zgxcv" podStartSLOduration=66.838789761 podStartE2EDuration="1m11.397595802s" totalImagesPullingTime="4.558806041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:31:05.396928904 +0000 UTC m=+71.963689456" watchObservedRunningTime="2026-09-08 11:31:05.397595802 +0000 UTC m=+71.964356337"
Sep 08 11:31:07.188115 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:07.188062    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:12.690409 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:12.690376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:13.991722 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:13.991670    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:14.092807 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:14.092776    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:27.147186 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:27.147045    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:27.216133 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:27.216084    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:31.700243 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:31.700201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:31:31.700625 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:31.700362    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 11:31:31.700625 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:31.700378    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c88c548cf-26prd: secret "image-registry-tls" not found
Sep 08 11:31:31.700625 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:31.700433    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls podName:1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d nodeName:}" failed. No retries permitted until 2026-09-08 11:32:35.700418271 +0000 UTC m=+162.267178794 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls") pod "image-registry-6c88c548cf-26prd" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d") : secret "image-registry-tls" not found
Sep 08 11:31:31.800737 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:31.800700    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:31:31.800931 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:31.800841    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 11:31:31.800931 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:31.800920    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls podName:1a77e471-aaa5-4f4d-932d-c8466b009a67 nodeName:}" failed. No retries permitted until 2026-09-08 11:32:35.800901964 +0000 UTC m=+162.367662483 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls") pod "dns-default-67xmf" (UID: "1a77e471-aaa5-4f4d-932d-c8466b009a67") : secret "dns-default-metrics-tls" not found
Sep 08 11:31:33.388076 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:33.388024    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:33.458816 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:33.458786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:36.365251 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:36.365217    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zgxcv"
Sep 08 11:31:38.421742 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:38.421709    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:43.495324 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.495288    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-w8mtk"]
Sep 08 11:31:43.499533 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.499519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-w8mtk"
Sep 08 11:31:43.501837 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.501810    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-kj64h\""
Sep 08 11:31:43.507757 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.507737    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-w8mtk"]
Sep 08 11:31:43.594189 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.594161    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tpxvv"]
Sep 08 11:31:43.597000 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.596977    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tpxvv"
Sep 08 11:31:43.599627 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.599608    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vg456\""
Sep 08 11:31:43.599821 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.599803    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 08 11:31:43.603194 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.600459    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 08 11:31:43.606346 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.606325    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tpxvv"]
Sep 08 11:31:43.683041 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.683005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtmzf\" (UniqueName: \"kubernetes.io/projected/ee2d3704-56d6-4097-b9a8-cc45fc2aaa17-kube-api-access-gtmzf\") pod \"network-check-source-685456655f-w8mtk\" (UID: \"ee2d3704-56d6-4097-b9a8-cc45fc2aaa17\") " pod="openshift-network-diagnostics/network-check-source-685456655f-w8mtk"
Sep 08 11:31:43.686272 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.686233    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:43.750515 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.750417    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:43.783858 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.783827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtmzf\" (UniqueName: \"kubernetes.io/projected/ee2d3704-56d6-4097-b9a8-cc45fc2aaa17-kube-api-access-gtmzf\") pod \"network-check-source-685456655f-w8mtk\" (UID: \"ee2d3704-56d6-4097-b9a8-cc45fc2aaa17\") " pod="openshift-network-diagnostics/network-check-source-685456655f-w8mtk"
Sep 08 11:31:43.783986 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.783879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5xg4\" (UniqueName: \"kubernetes.io/projected/110b167e-aed6-4df1-8c00-9aa826338eca-kube-api-access-l5xg4\") pod \"volume-data-source-validator-695c895bd6-tpxvv\" (UID: \"110b167e-aed6-4df1-8c00-9aa826338eca\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tpxvv"
Sep 08 11:31:43.793472 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.793450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtmzf\" (UniqueName: \"kubernetes.io/projected/ee2d3704-56d6-4097-b9a8-cc45fc2aaa17-kube-api-access-gtmzf\") pod \"network-check-source-685456655f-w8mtk\" (UID: \"ee2d3704-56d6-4097-b9a8-cc45fc2aaa17\") " pod="openshift-network-diagnostics/network-check-source-685456655f-w8mtk"
Sep 08 11:31:43.808389 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.808358    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-w8mtk"
Sep 08 11:31:43.885004 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.884972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5xg4\" (UniqueName: \"kubernetes.io/projected/110b167e-aed6-4df1-8c00-9aa826338eca-kube-api-access-l5xg4\") pod \"volume-data-source-validator-695c895bd6-tpxvv\" (UID: \"110b167e-aed6-4df1-8c00-9aa826338eca\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tpxvv"
Sep 08 11:31:43.896163 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.896136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5xg4\" (UniqueName: \"kubernetes.io/projected/110b167e-aed6-4df1-8c00-9aa826338eca-kube-api-access-l5xg4\") pod \"volume-data-source-validator-695c895bd6-tpxvv\" (UID: \"110b167e-aed6-4df1-8c00-9aa826338eca\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tpxvv"
Sep 08 11:31:43.909096 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:43.909059    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tpxvv"
Sep 08 11:31:44.078551 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:44.078486    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-w8mtk"]
Sep 08 11:31:44.081011 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:31:44.080984    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2d3704_56d6_4097_b9a8_cc45fc2aaa17.slice/crio-07b19da32c305da767ec1f74d2d89c4eef11c5e18a80ce24b054251d1e016707 WatchSource:0}: Error finding container 07b19da32c305da767ec1f74d2d89c4eef11c5e18a80ce24b054251d1e016707: Status 404 returned error can't find the container with id 07b19da32c305da767ec1f74d2d89c4eef11c5e18a80ce24b054251d1e016707
Sep 08 11:31:44.178854 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:44.178811    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tpxvv"]
Sep 08 11:31:44.181443 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:31:44.181421    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110b167e_aed6_4df1_8c00_9aa826338eca.slice/crio-1e271ceef3e4a47a630ecc0adaeade6592c3a6e391b70f772c7b6754cba30f75 WatchSource:0}: Error finding container 1e271ceef3e4a47a630ecc0adaeade6592c3a6e391b70f772c7b6754cba30f75: Status 404 returned error can't find the container with id 1e271ceef3e4a47a630ecc0adaeade6592c3a6e391b70f772c7b6754cba30f75
Sep 08 11:31:44.452365 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:44.452267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-w8mtk" event={"ID":"ee2d3704-56d6-4097-b9a8-cc45fc2aaa17","Type":"ContainerStarted","Data":"ff75a6a39061990f00a058beb3752e89a87fcd8c68c5954bb1fc988989ba2371"}
Sep 08 11:31:44.452365 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:44.452326    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-w8mtk" event={"ID":"ee2d3704-56d6-4097-b9a8-cc45fc2aaa17","Type":"ContainerStarted","Data":"07b19da32c305da767ec1f74d2d89c4eef11c5e18a80ce24b054251d1e016707"}
Sep 08 11:31:44.453393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:44.453370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tpxvv" event={"ID":"110b167e-aed6-4df1-8c00-9aa826338eca","Type":"ContainerStarted","Data":"1e271ceef3e4a47a630ecc0adaeade6592c3a6e391b70f772c7b6754cba30f75"}
Sep 08 11:31:44.470622 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:44.470579    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-w8mtk" podStartSLOduration=1.470562 podStartE2EDuration="1.470562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:31:44.469858357 +0000 UTC m=+111.036618899" watchObservedRunningTime="2026-09-08 11:31:44.470562 +0000 UTC m=+111.037322543"
Sep 08 11:31:45.457507 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:45.457456    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tpxvv" event={"ID":"110b167e-aed6-4df1-8c00-9aa826338eca","Type":"ContainerStarted","Data":"29cd07dfb6d85976712f20ba24ccf3be82736ccb99c3956f1d20d058e6767a20"}
Sep 08 11:31:45.485103 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:45.485053    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-tpxvv" podStartSLOduration=1.562629202 podStartE2EDuration="2.485035582s" totalImagesPullingTime="922.40638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:31:45.484268367 +0000 UTC m=+112.051028908" watchObservedRunningTime="2026-09-08 11:31:45.485035582 +0000 UTC m=+112.051796125"
Sep 08 11:31:45.613255 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:45.613230    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:45.705212 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:45.705175    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:49.071696 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.071658    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dclqn"]
Sep 08 11:31:49.074511 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.074496    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.076988 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.076950    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 11:31:49.078026 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.078004    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 11:31:49.078121 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.078022    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vrknd\""
Sep 08 11:31:49.078121 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.078036    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 11:31:49.078121 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.078011    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 11:31:49.087236 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.087214    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dclqn"]
Sep 08 11:31:49.221617 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.221573    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-crio-socket\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.221772 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.221633    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-data-volume\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.221772 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.221674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.221772 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.221692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.221772 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.221748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st8pk\" (UniqueName: \"kubernetes.io/projected/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-api-access-st8pk\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.322280 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.322216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-data-volume\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.322280 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.322256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.322280 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.322273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.322492 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.322313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st8pk\" (UniqueName: \"kubernetes.io/projected/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-api-access-st8pk\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.322492 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:49.322372    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 11:31:49.322492 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:49.322406    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 11:31:49.322492 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.322430    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-crio-socket\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.322492 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.322374    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-crio-socket\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.322492 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:49.322455    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm podName:8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f nodeName:}" failed. No retries permitted until 2026-09-08 11:31:49.822434214 +0000 UTC m=+116.389194760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dclqn" (UID: "8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f") : configmap "kube-rbac-proxy" not found
Sep 08 11:31:49.322791 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:49.322511    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls podName:8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f nodeName:}" failed. No retries permitted until 2026-09-08 11:31:49.822499516 +0000 UTC m=+116.389260036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dclqn" (UID: "8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f") : secret "insights-runtime-extractor-tls" not found
Sep 08 11:31:49.322791 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.322559    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-data-volume\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.332877 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.332857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st8pk\" (UniqueName: \"kubernetes.io/projected/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-api-access-st8pk\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.413963 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.413939    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:49.513551 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.513531    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:49.825275 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.825235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.825542 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:49.825284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:49.825542 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:49.825361    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 11:31:49.825542 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:49.825392    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 11:31:49.825542 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:49.825426    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm podName:8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f nodeName:}" failed. No retries permitted until 2026-09-08 11:31:50.825411276 +0000 UTC m=+117.392171795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dclqn" (UID: "8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f") : configmap "kube-rbac-proxy" not found
Sep 08 11:31:49.825542 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:49.825447    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls podName:8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f nodeName:}" failed. No retries permitted until 2026-09-08 11:31:50.825432215 +0000 UTC m=+117.392192751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dclqn" (UID: "8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f") : secret "insights-runtime-extractor-tls" not found
Sep 08 11:31:50.281091 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:50.281058    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:50.380717 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:50.380677    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:50.832987 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:50.832956    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:50.832987 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:50.832988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:50.833169 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:50.833073    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 11:31:50.833169 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:50.833096    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 11:31:50.833169 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:50.833128    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm podName:8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f nodeName:}" failed. No retries permitted until 2026-09-08 11:31:52.833115061 +0000 UTC m=+119.399875606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dclqn" (UID: "8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f") : configmap "kube-rbac-proxy" not found
Sep 08 11:31:50.833169 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:50.833144    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls podName:8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f nodeName:}" failed. No retries permitted until 2026-09-08 11:31:52.833135463 +0000 UTC m=+119.399895982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dclqn" (UID: "8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f") : secret "insights-runtime-extractor-tls" not found
Sep 08 11:31:51.005042 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.005010    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-7d47p"]
Sep 08 11:31:51.007951 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.007936    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-7d47p"
Sep 08 11:31:51.010527 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.010502    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 11:31:51.010527 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.010523    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 11:31:51.010703 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.010540    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 11:31:51.011712 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.011691    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8vv2r\""
Sep 08 11:31:51.011712 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.011712    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 11:31:51.020520 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.020501    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-7d47p"]
Sep 08 11:31:51.054350 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.054329    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:51.130424 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.130370    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:51.136419 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.136401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6f572018-7b81-4b57-9232-81c1b54b1bd7-signing-key\") pod \"service-ca-75d99c484-7d47p\" (UID: \"6f572018-7b81-4b57-9232-81c1b54b1bd7\") " pod="openshift-service-ca/service-ca-75d99c484-7d47p"
Sep 08 11:31:51.136485 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.136428    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb87x\" (UniqueName: \"kubernetes.io/projected/6f572018-7b81-4b57-9232-81c1b54b1bd7-kube-api-access-jb87x\") pod \"service-ca-75d99c484-7d47p\" (UID: \"6f572018-7b81-4b57-9232-81c1b54b1bd7\") " pod="openshift-service-ca/service-ca-75d99c484-7d47p"
Sep 08 11:31:51.136528 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.136497    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6f572018-7b81-4b57-9232-81c1b54b1bd7-signing-cabundle\") pod \"service-ca-75d99c484-7d47p\" (UID: \"6f572018-7b81-4b57-9232-81c1b54b1bd7\") " pod="openshift-service-ca/service-ca-75d99c484-7d47p"
Sep 08 11:31:51.237391 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.237359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6f572018-7b81-4b57-9232-81c1b54b1bd7-signing-cabundle\") pod \"service-ca-75d99c484-7d47p\" (UID: \"6f572018-7b81-4b57-9232-81c1b54b1bd7\") " pod="openshift-service-ca/service-ca-75d99c484-7d47p"
Sep 08 11:31:51.237531 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.237457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6f572018-7b81-4b57-9232-81c1b54b1bd7-signing-key\") pod \"service-ca-75d99c484-7d47p\" (UID: \"6f572018-7b81-4b57-9232-81c1b54b1bd7\") " pod="openshift-service-ca/service-ca-75d99c484-7d47p"
Sep 08 11:31:51.237531 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.237499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb87x\" (UniqueName: \"kubernetes.io/projected/6f572018-7b81-4b57-9232-81c1b54b1bd7-kube-api-access-jb87x\") pod \"service-ca-75d99c484-7d47p\" (UID: \"6f572018-7b81-4b57-9232-81c1b54b1bd7\") " pod="openshift-service-ca/service-ca-75d99c484-7d47p"
Sep 08 11:31:51.238252 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.238231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6f572018-7b81-4b57-9232-81c1b54b1bd7-signing-cabundle\") pod \"service-ca-75d99c484-7d47p\" (UID: \"6f572018-7b81-4b57-9232-81c1b54b1bd7\") " pod="openshift-service-ca/service-ca-75d99c484-7d47p"
Sep 08 11:31:51.239966 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.239941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6f572018-7b81-4b57-9232-81c1b54b1bd7-signing-key\") pod \"service-ca-75d99c484-7d47p\" (UID: \"6f572018-7b81-4b57-9232-81c1b54b1bd7\") " pod="openshift-service-ca/service-ca-75d99c484-7d47p"
Sep 08 11:31:51.247268 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.247233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb87x\" (UniqueName: \"kubernetes.io/projected/6f572018-7b81-4b57-9232-81c1b54b1bd7-kube-api-access-jb87x\") pod \"service-ca-75d99c484-7d47p\" (UID: \"6f572018-7b81-4b57-9232-81c1b54b1bd7\") " pod="openshift-service-ca/service-ca-75d99c484-7d47p"
Sep 08 11:31:51.316443 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.316420    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-7d47p"
Sep 08 11:31:51.590414 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:51.590381    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-7d47p"]
Sep 08 11:31:51.593411 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:31:51.593387    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f572018_7b81_4b57_9232_81c1b54b1bd7.slice/crio-cfc850f6011f9fc0bc7b776a70e15692ba8a8a1a0e26093d9cf5c7de3ce4bd90 WatchSource:0}: Error finding container cfc850f6011f9fc0bc7b776a70e15692ba8a8a1a0e26093d9cf5c7de3ce4bd90: Status 404 returned error can't find the container with id cfc850f6011f9fc0bc7b776a70e15692ba8a8a1a0e26093d9cf5c7de3ce4bd90
Sep 08 11:31:52.030605 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:52.030577    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l76h7_00567f83-0336-467f-826b-a57e063c76ac/dns-node-resolver/0.log"
Sep 08 11:31:52.475344 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:52.475305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-7d47p" event={"ID":"6f572018-7b81-4b57-9232-81c1b54b1bd7","Type":"ContainerStarted","Data":"cfc850f6011f9fc0bc7b776a70e15692ba8a8a1a0e26093d9cf5c7de3ce4bd90"}
Sep 08 11:31:52.630610 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:52.630578    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5xw68_838a5800-8a91-4b1b-bc39-655796992d82/node-ca/0.log"
Sep 08 11:31:52.757605 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:52.757532    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:52.851754 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:52.851716    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:52.851754 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:52.851762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:52.851953 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:52.851852    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 11:31:52.851953 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:52.851889    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 11:31:52.851953 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:52.851920    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm podName:8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f nodeName:}" failed. No retries permitted until 2026-09-08 11:31:56.851903907 +0000 UTC m=+123.418664449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dclqn" (UID: "8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f") : configmap "kube-rbac-proxy" not found
Sep 08 11:31:52.851953 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:52.851935    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls podName:8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f nodeName:}" failed. No retries permitted until 2026-09-08 11:31:56.851928173 +0000 UTC m=+123.418688692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dclqn" (UID: "8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f") : secret "insights-runtime-extractor-tls" not found
Sep 08 11:31:53.430899 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:53.430878    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b5kmv_c229e7b3-1c76-407e-b4cf-6208554a0009/serve-healthcheck-canary/0.log"
Sep 08 11:31:53.480126 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:53.480096    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-7d47p" event={"ID":"6f572018-7b81-4b57-9232-81c1b54b1bd7","Type":"ContainerStarted","Data":"4a380894e20a3aa95e80157f361ed282d88578ec217891b360f35dc7f60ca439"}
Sep 08 11:31:53.498865 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:53.498708    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-7d47p" podStartSLOduration=1.739702398 podStartE2EDuration="3.498692169s" totalImagesPullingTime="1.758989771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:31:53.498582868 +0000 UTC m=+120.065343408" watchObservedRunningTime="2026-09-08 11:31:53.498692169 +0000 UTC m=+120.065452711"
Sep 08 11:31:54.542736 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:54.542667    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:54.729688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:54.729634    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:54.830774 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:54.830701    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:54.963538 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:54.963406    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:55.214936 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:55.214903    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:56.013317 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:56.013268    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:56.396690 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:56.396608    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:56.797205 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:56.797144    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:56.885679 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:56.885655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:56.885813 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:56.885686    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:31:56.885813 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:56.885768    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 11:31:56.885813 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:56.885783    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 11:31:56.885908 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:56.885824    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm podName:8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f nodeName:}" failed. No retries permitted until 2026-09-08 11:32:04.885808035 +0000 UTC m=+131.452568559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dclqn" (UID: "8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f") : configmap "kube-rbac-proxy" not found
Sep 08 11:31:56.885908 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:31:56.885846    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls podName:8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f nodeName:}" failed. No retries permitted until 2026-09-08 11:32:04.885835278 +0000 UTC m=+131.452595796 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dclqn" (UID: "8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f") : secret "insights-runtime-extractor-tls" not found
Sep 08 11:31:56.989914 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:56.989889    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:57.605923 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:57.605887    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:57.996535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:57.996503    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:59.178637 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:59.178601    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:59.282223 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:59.282171    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:31:59.593192 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:31:59.593163    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:00.010323 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:00.010290    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:00.420372 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:00.420289    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:00.773534 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:00.773503    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:00.873707 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:00.873677    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:01.001931 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:01.001900    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:01.197496 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:01.197404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:01.393272 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:01.393238    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:01.992695 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:01.992660    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:02.193240 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:02.193206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:02.789138 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:02.789101    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:03.353496 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:03.353453    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:03.448511 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:03.448481    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:03.836691 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:03.836660    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:32:03.838988 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:03.838966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a72df4f-cfd6-4867-92ee-7f559401f9d0-metrics-certs\") pod \"network-metrics-daemon-q2z6p\" (UID: \"3a72df4f-cfd6-4867-92ee-7f559401f9d0\") " pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:32:03.969599 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:03.969574    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt4h2\""
Sep 08 11:32:03.977590 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:03.977573    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2z6p"
Sep 08 11:32:04.000479 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.000444    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:04.245739 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.245707    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q2z6p"]
Sep 08 11:32:04.248566 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:32:04.248530    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a72df4f_cfd6_4867_92ee_7f559401f9d0.slice/crio-fa13920bbb0b82b877635811a5badbc975859adcc56117d6d1e3bbee96f53a55 WatchSource:0}: Error finding container fa13920bbb0b82b877635811a5badbc975859adcc56117d6d1e3bbee96f53a55: Status 404 returned error can't find the container with id fa13920bbb0b82b877635811a5badbc975859adcc56117d6d1e3bbee96f53a55
Sep 08 11:32:04.388777 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.388744    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:04.511574 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.511498    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q2z6p" event={"ID":"3a72df4f-cfd6-4867-92ee-7f559401f9d0","Type":"ContainerStarted","Data":"fa13920bbb0b82b877635811a5badbc975859adcc56117d6d1e3bbee96f53a55"}
Sep 08 11:32:04.667055 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.667020    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:04.767631 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.767559    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:04.868047 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.868016    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:04.945384 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.945353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:32:04.945576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.945390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:32:04.946001 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.945968    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:32:04.947841 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.947820    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dclqn\" (UID: \"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f\") " pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:32:04.985540 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.985512    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vrknd\""
Sep 08 11:32:04.992168 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.992142    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:04.992999 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:04.992978    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dclqn"
Sep 08 11:32:05.184218 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:05.184190    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:05.439623 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:05.439502    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dclqn"]
Sep 08 11:32:05.441992 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:32:05.441958    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc57d0a_1e2d_43ce_a7a3_471bb5952c9f.slice/crio-cadd9834f2bc20be7d8d84973e40af46aaa3566a1e197e912def9b53c8607d72 WatchSource:0}: Error finding container cadd9834f2bc20be7d8d84973e40af46aaa3566a1e197e912def9b53c8607d72: Status 404 returned error can't find the container with id cadd9834f2bc20be7d8d84973e40af46aaa3566a1e197e912def9b53c8607d72
Sep 08 11:32:05.516966 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:05.516936    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q2z6p" event={"ID":"3a72df4f-cfd6-4867-92ee-7f559401f9d0","Type":"ContainerStarted","Data":"81a5e0d0152e6e95be1a29b46482bfe1e2cd698ab2945a95549388e226255699"}
Sep 08 11:32:05.517117 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:05.516996    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q2z6p" event={"ID":"3a72df4f-cfd6-4867-92ee-7f559401f9d0","Type":"ContainerStarted","Data":"a7cfb8f5ad97a92eb8d39150bba35f389c625f28b655af351d3a080e107c5cc2"}
Sep 08 11:32:05.518528 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:05.518509    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dclqn" event={"ID":"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f","Type":"ContainerStarted","Data":"1c6a9605e3eec55af9bf9ae9578b7c5af9127e1c106ee5374af6bf0a2a94fa3f"}
Sep 08 11:32:05.518620 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:05.518549    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dclqn" event={"ID":"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f","Type":"ContainerStarted","Data":"cadd9834f2bc20be7d8d84973e40af46aaa3566a1e197e912def9b53c8607d72"}
Sep 08 11:32:05.535757 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:05.535713    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q2z6p" podStartSLOduration=130.55596164 podStartE2EDuration="2m11.53569802s" totalImagesPullingTime="979.73638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:05.535216621 +0000 UTC m=+132.101977164" watchObservedRunningTime="2026-09-08 11:32:05.53569802 +0000 UTC m=+132.102458552"
Sep 08 11:32:06.523683 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:06.523651    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dclqn" event={"ID":"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f","Type":"ContainerStarted","Data":"9012e16701ee3a5b6ce5d2d9ba268b020eff36d62371651afac84c615a8dfe72"}
Sep 08 11:32:07.878443 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:07.878339    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:07.975581 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:07.975546    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:08.082862 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:08.082836    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:08.180193 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:08.180118    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:08.527561 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:08.527529    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:08.531088 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:08.531062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dclqn" event={"ID":"8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f","Type":"ContainerStarted","Data":"e2d8bece4a163758c3e0f5f5cd803ca75bf314e96f59d3d40b3e07d80b96ed8a"}
Sep 08 11:32:08.551028 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:08.550989    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dclqn" podStartSLOduration=17.619028214 podStartE2EDuration="19.550976883s" totalImagesPullingTime="1.931948669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:31:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:08.549553418 +0000 UTC m=+135.116313958" watchObservedRunningTime="2026-09-08 11:32:08.550976883 +0000 UTC m=+135.117737424"
Sep 08 11:32:08.599446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:08.599421    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:08.725063 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:08.725033    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:08.793301 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:08.793242    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:09.130630 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:09.130554    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:09.202377 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:09.202338    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:09.743682 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:09.743649    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:09.944598 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:09.944556    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:10.285026 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:10.284996    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:10.566564 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:10.566491    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:10.641198 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:10.641173    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:15.856686 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.856656    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"]
Sep 08 11:32:15.859681 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.859654    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"
Sep 08 11:32:15.859820 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.859662    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-jdgsc"]
Sep 08 11:32:15.863209 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.863187    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-jdgsc"
Sep 08 11:32:15.863303 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.863273    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 11:32:15.863303 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.863287    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 11:32:15.863419 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.863279    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 11:32:15.863510 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.863457    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 11:32:15.865732 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.865705    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 08 11:32:15.866361 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.866345    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 08 11:32:15.866451 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.866362    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-24wpv\""
Sep 08 11:32:15.871971 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.871935    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"]
Sep 08 11:32:15.875010 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.874990    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"]
Sep 08 11:32:15.878003 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.877987    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:15.880452 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.880432    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 11:32:15.880598 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.880433    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 11:32:15.880659 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.880635    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 11:32:15.880885 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.880868    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 11:32:15.881116 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.881098    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-jdgsc"]
Sep 08 11:32:15.892647 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.892631    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"]
Sep 08 11:32:15.933338 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.933313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f890716-0fb5-4950-96e2-dab6f19b24ff-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:15.933430 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.933343    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/083f7738-88a3-4d4a-8ba4-0a07ce60dadc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c686fc48b-6xcml\" (UID: \"083f7738-88a3-4d4a-8ba4-0a07ce60dadc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"
Sep 08 11:32:15.933430 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.933361    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/083f7738-88a3-4d4a-8ba4-0a07ce60dadc-tmp\") pod \"klusterlet-addon-workmgr-7c686fc48b-6xcml\" (UID: \"083f7738-88a3-4d4a-8ba4-0a07ce60dadc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"
Sep 08 11:32:15.933532 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.933426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8f890716-0fb5-4950-96e2-dab6f19b24ff-hub\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:15.933532 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.933493    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sxkr\" (UniqueName: \"kubernetes.io/projected/083f7738-88a3-4d4a-8ba4-0a07ce60dadc-kube-api-access-9sxkr\") pod \"klusterlet-addon-workmgr-7c686fc48b-6xcml\" (UID: \"083f7738-88a3-4d4a-8ba4-0a07ce60dadc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"
Sep 08 11:32:15.933532 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.933517    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8f890716-0fb5-4950-96e2-dab6f19b24ff-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:15.933659 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.933536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6d98443-0a7e-403f-9884-c92a00056280-nginx-conf\") pod \"networking-console-plugin-79977f9844-jdgsc\" (UID: \"b6d98443-0a7e-403f-9884-c92a00056280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jdgsc"
Sep 08 11:32:15.933659 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.933554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfmz9\" (UniqueName: \"kubernetes.io/projected/8f890716-0fb5-4950-96e2-dab6f19b24ff-kube-api-access-cfmz9\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:15.933659 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.933645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8f890716-0fb5-4950-96e2-dab6f19b24ff-ca\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:15.933757 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.933679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d98443-0a7e-403f-9884-c92a00056280-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jdgsc\" (UID: \"b6d98443-0a7e-403f-9884-c92a00056280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jdgsc"
Sep 08 11:32:15.933757 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:15.933719    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f890716-0fb5-4950-96e2-dab6f19b24ff-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.035068 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f890716-0fb5-4950-96e2-dab6f19b24ff-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.035068 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/083f7738-88a3-4d4a-8ba4-0a07ce60dadc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c686fc48b-6xcml\" (UID: \"083f7738-88a3-4d4a-8ba4-0a07ce60dadc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"
Sep 08 11:32:16.035284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035086    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/083f7738-88a3-4d4a-8ba4-0a07ce60dadc-tmp\") pod \"klusterlet-addon-workmgr-7c686fc48b-6xcml\" (UID: \"083f7738-88a3-4d4a-8ba4-0a07ce60dadc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"
Sep 08 11:32:16.035284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035115    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8f890716-0fb5-4950-96e2-dab6f19b24ff-hub\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.035284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sxkr\" (UniqueName: \"kubernetes.io/projected/083f7738-88a3-4d4a-8ba4-0a07ce60dadc-kube-api-access-9sxkr\") pod \"klusterlet-addon-workmgr-7c686fc48b-6xcml\" (UID: \"083f7738-88a3-4d4a-8ba4-0a07ce60dadc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"
Sep 08 11:32:16.035284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8f890716-0fb5-4950-96e2-dab6f19b24ff-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.035284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035184    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6d98443-0a7e-403f-9884-c92a00056280-nginx-conf\") pod \"networking-console-plugin-79977f9844-jdgsc\" (UID: \"b6d98443-0a7e-403f-9884-c92a00056280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jdgsc"
Sep 08 11:32:16.035284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfmz9\" (UniqueName: \"kubernetes.io/projected/8f890716-0fb5-4950-96e2-dab6f19b24ff-kube-api-access-cfmz9\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.035284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035226    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8f890716-0fb5-4950-96e2-dab6f19b24ff-ca\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.035284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035251    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d98443-0a7e-403f-9884-c92a00056280-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jdgsc\" (UID: \"b6d98443-0a7e-403f-9884-c92a00056280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jdgsc"
Sep 08 11:32:16.035721 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f890716-0fb5-4950-96e2-dab6f19b24ff-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.035721 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035541    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/083f7738-88a3-4d4a-8ba4-0a07ce60dadc-tmp\") pod \"klusterlet-addon-workmgr-7c686fc48b-6xcml\" (UID: \"083f7738-88a3-4d4a-8ba4-0a07ce60dadc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"
Sep 08 11:32:16.035939 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035908    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f890716-0fb5-4950-96e2-dab6f19b24ff-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.036037 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.035949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6d98443-0a7e-403f-9884-c92a00056280-nginx-conf\") pod \"networking-console-plugin-79977f9844-jdgsc\" (UID: \"b6d98443-0a7e-403f-9884-c92a00056280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jdgsc"
Sep 08 11:32:16.038030 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.037998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8f890716-0fb5-4950-96e2-dab6f19b24ff-ca\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.038030 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.038025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d98443-0a7e-403f-9884-c92a00056280-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jdgsc\" (UID: \"b6d98443-0a7e-403f-9884-c92a00056280\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jdgsc"
Sep 08 11:32:16.038230 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.038211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8f890716-0fb5-4950-96e2-dab6f19b24ff-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.038291 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.038271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8f890716-0fb5-4950-96e2-dab6f19b24ff-hub\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.038539 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.038520    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f890716-0fb5-4950-96e2-dab6f19b24ff-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.038637 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.038618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/083f7738-88a3-4d4a-8ba4-0a07ce60dadc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c686fc48b-6xcml\" (UID: \"083f7738-88a3-4d4a-8ba4-0a07ce60dadc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"
Sep 08 11:32:16.049056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.049030    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfmz9\" (UniqueName: \"kubernetes.io/projected/8f890716-0fb5-4950-96e2-dab6f19b24ff-kube-api-access-cfmz9\") pod \"cluster-proxy-proxy-agent-69b4dfc4fd-8l84h\" (UID: \"8f890716-0fb5-4950-96e2-dab6f19b24ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.053029 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.053005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sxkr\" (UniqueName: \"kubernetes.io/projected/083f7738-88a3-4d4a-8ba4-0a07ce60dadc-kube-api-access-9sxkr\") pod \"klusterlet-addon-workmgr-7c686fc48b-6xcml\" (UID: \"083f7738-88a3-4d4a-8ba4-0a07ce60dadc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"
Sep 08 11:32:16.170543 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.170450    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"
Sep 08 11:32:16.175523 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.175504    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-jdgsc"
Sep 08 11:32:16.200319 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.200287    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"
Sep 08 11:32:16.452177 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.452102    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"]
Sep 08 11:32:16.454861 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:32:16.454824    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083f7738_88a3_4d4a_8ba4_0a07ce60dadc.slice/crio-39e7959ae68d6f4e6cddc88c9057350da5254c9b4c71f9b8530e1be06761c544 WatchSource:0}: Error finding container 39e7959ae68d6f4e6cddc88c9057350da5254c9b4c71f9b8530e1be06761c544: Status 404 returned error can't find the container with id 39e7959ae68d6f4e6cddc88c9057350da5254c9b4c71f9b8530e1be06761c544
Sep 08 11:32:16.458303 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.458281    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-jdgsc"]
Sep 08 11:32:16.460802 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:32:16.460777    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d98443_0a7e_403f_9884_c92a00056280.slice/crio-1690cfcf587af1da6fe7b597e3b64cfd2cd57333be074f106cd086fb9661a750 WatchSource:0}: Error finding container 1690cfcf587af1da6fe7b597e3b64cfd2cd57333be074f106cd086fb9661a750: Status 404 returned error can't find the container with id 1690cfcf587af1da6fe7b597e3b64cfd2cd57333be074f106cd086fb9661a750
Sep 08 11:32:16.495094 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.495070    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h"]
Sep 08 11:32:16.498450 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:32:16.498429    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f890716_0fb5_4950_96e2_dab6f19b24ff.slice/crio-a5d12ed4a656ee62c2699ef70e52b287acdbd24af71cb946abb09010c9211137 WatchSource:0}: Error finding container a5d12ed4a656ee62c2699ef70e52b287acdbd24af71cb946abb09010c9211137: Status 404 returned error can't find the container with id a5d12ed4a656ee62c2699ef70e52b287acdbd24af71cb946abb09010c9211137
Sep 08 11:32:16.555524 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.555489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h" event={"ID":"8f890716-0fb5-4950-96e2-dab6f19b24ff","Type":"ContainerStarted","Data":"a5d12ed4a656ee62c2699ef70e52b287acdbd24af71cb946abb09010c9211137"}
Sep 08 11:32:16.556491 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.556451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-jdgsc" event={"ID":"b6d98443-0a7e-403f-9884-c92a00056280","Type":"ContainerStarted","Data":"1690cfcf587af1da6fe7b597e3b64cfd2cd57333be074f106cd086fb9661a750"}
Sep 08 11:32:16.557496 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:16.557461    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml" event={"ID":"083f7738-88a3-4d4a-8ba4-0a07ce60dadc","Type":"ContainerStarted","Data":"39e7959ae68d6f4e6cddc88c9057350da5254c9b4c71f9b8530e1be06761c544"}
Sep 08 11:32:17.135189 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:17.135149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:17.236737 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:17.236706    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:17.995178 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:17.995141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:18.098789 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:18.098708    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:18.183877 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:18.183694    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:18.396411 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:18.396314    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:22.587453 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:22.587430    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-jdgsc" event={"ID":"b6d98443-0a7e-403f-9884-c92a00056280","Type":"ContainerStarted","Data":"cae8c3ea2be4cb5a343aa8eeb6c01782a93c84c3042acaa70cf9220c2243355f"}
Sep 08 11:32:22.589438 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:22.589401    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml" event={"ID":"083f7738-88a3-4d4a-8ba4-0a07ce60dadc","Type":"ContainerStarted","Data":"47b37e45c1faa4d6f46d58018f57b28e2e7fd6dd70bb6ee507364098b2c38794"}
Sep 08 11:32:22.589615 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:22.589600    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"
Sep 08 11:32:22.591130 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:22.591083    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml"
Sep 08 11:32:22.591234 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:22.591202    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h" event={"ID":"8f890716-0fb5-4950-96e2-dab6f19b24ff","Type":"ContainerStarted","Data":"aa4b6cd77132cac82b6c06305f8eae2b2eec63265ff7211936676bf9b53f5833"}
Sep 08 11:32:22.615055 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:22.615020    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:22.615576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:22.615535    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-jdgsc" podStartSLOduration=1.629842355 podStartE2EDuration="7.615519501s" totalImagesPullingTime="5.985677146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:22.613985417 +0000 UTC m=+149.180745958" watchObservedRunningTime="2026-09-08 11:32:22.615519501 +0000 UTC m=+149.182280042"
Sep 08 11:32:22.638028 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:22.637859    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c686fc48b-6xcml" podStartSLOduration=1.6299993480000001 podStartE2EDuration="7.637845899s" totalImagesPullingTime="6.007846551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:22.636945164 +0000 UTC m=+149.203705704" watchObservedRunningTime="2026-09-08 11:32:22.637845899 +0000 UTC m=+149.204606438"
Sep 08 11:32:22.773068 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:22.773039    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:22.873437 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:22.873396    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:23.299870 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:23.299833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:23.375608 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:23.375572    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:23.596945 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:23.596861    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h" event={"ID":"8f890716-0fb5-4950-96e2-dab6f19b24ff","Type":"ContainerStarted","Data":"a463ade58957bf9b71b02324f96afe652f152ddf1879ae84639c133e53af9e03"}
Sep 08 11:32:23.596945 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:23.596907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h" event={"ID":"8f890716-0fb5-4950-96e2-dab6f19b24ff","Type":"ContainerStarted","Data":"4724c216b2e0be863519a0bfbe19d9fcbb8f4ecd3ffb21c0a8d42f8b3e0b9298"}
Sep 08 11:32:23.619453 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:23.619409    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b4dfc4fd-8l84h" podStartSLOduration=2.652907948 podStartE2EDuration="8.619396242s" totalImagesPullingTime="5.966488294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:23.618496556 +0000 UTC m=+150.185257094" watchObservedRunningTime="2026-09-08 11:32:23.619396242 +0000 UTC m=+150.186156782"
Sep 08 11:32:24.195979 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:24.195934    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:24.279999 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:24.279958    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:24.368850 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:24.368821    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:24.844633 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:24.844601    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:24.945360 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:24.945298    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:26.356354 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:26.356296    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:26.456507 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:26.456458    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:28.379329 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:28.379298    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:28.486432 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:28.486404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:29.335283 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.335252    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:29.590703 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.590595    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:29.648945 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.648911    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kwbt2"]
Sep 08 11:32:29.654442 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.654423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.657321 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.657302    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 11:32:29.657674 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.657648    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 11:32:29.657758 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.657738    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 11:32:29.658548 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.658525    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 11:32:29.658661 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.658643    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k7x8b\""
Sep 08 11:32:29.658762 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.658746    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 11:32:29.658847 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.658832    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 11:32:29.699365 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.699329    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:29.745455 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.745421    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtl5v\" (UniqueName: \"kubernetes.io/projected/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-kube-api-access-wtl5v\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.745654 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.745495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-metrics-client-ca\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.745654 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.745532    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-sys\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.745654 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.745558    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-accelerators-collector-config\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.745778 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.745649    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-tls\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.745778 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.745714    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-textfile\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.745778 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.745763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-wtmp\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.745884 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.745787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-root\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.745884 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.745848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.803088 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.803059    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:29.846603 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.846526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-sys\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.846603 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.846569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-accelerators-collector-config\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.846806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.846621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-tls\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.846806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.846632    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-sys\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.846806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.846655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-textfile\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.846806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.846698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-wtmp\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.846806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.846723    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-root\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.846806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.846754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.846806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.846779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtl5v\" (UniqueName: \"kubernetes.io/projected/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-kube-api-access-wtl5v\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.847149 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.846818    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-metrics-client-ca\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.847149 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.846833    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-wtmp\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.847149 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.846833    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-root\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.847149 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.846998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-textfile\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.847579 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.847558    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-accelerators-collector-config\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.847662 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.847574    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-metrics-client-ca\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.848979 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.848964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-tls\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.849046 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.848994    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.859099 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.859079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtl5v\" (UniqueName: \"kubernetes.io/projected/8614060e-29d7-4d4a-b4a0-a3c52cc2caf5-kube-api-access-wtl5v\") pod \"node-exporter-kwbt2\" (UID: \"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5\") " pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.903234 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.903203    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:29.964205 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:29.964177    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kwbt2"
Sep 08 11:32:29.975984 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:32:29.975954    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8614060e_29d7_4d4a_b4a0_a3c52cc2caf5.slice/crio-cf2a41a7dd828c8fb3b0aeb6381ebe88a71dc519e18d926b14e7e16bdd7032b3 WatchSource:0}: Error finding container cf2a41a7dd828c8fb3b0aeb6381ebe88a71dc519e18d926b14e7e16bdd7032b3: Status 404 returned error can't find the container with id cf2a41a7dd828c8fb3b0aeb6381ebe88a71dc519e18d926b14e7e16bdd7032b3
Sep 08 11:32:30.003076 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:30.003053    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:30.307389 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:30.307360    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:30.537120 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:30.537090    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:30.618525 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:30.618399    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwbt2" event={"ID":"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5","Type":"ContainerStarted","Data":"cf2a41a7dd828c8fb3b0aeb6381ebe88a71dc519e18d926b14e7e16bdd7032b3"}
Sep 08 11:32:30.638537 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:30.638509    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:30.739907 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:30.739874    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:30.839577 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:32:30.839542    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6c88c548cf-26prd" podUID="1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d"
Sep 08 11:32:30.850676 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:32:30.850650    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-67xmf" podUID="1a77e471-aaa5-4f4d-932d-c8466b009a67"
Sep 08 11:32:31.037441 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:31.037400    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:31.621885 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:31.621857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-67xmf"
Sep 08 11:32:31.622421 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:31.621857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:32:32.627713 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:32.627627    2561 generic.go:356] "Generic (PLEG): container finished" podID="8614060e-29d7-4d4a-b4a0-a3c52cc2caf5" containerID="958debf8ce6e9afc17c8fc98c42daaaeb2d280a72f594fc6f57b66e1ccf039fa" exitCode=0
Sep 08 11:32:32.628140 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:32.627715    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwbt2" event={"ID":"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5","Type":"ContainerDied","Data":"958debf8ce6e9afc17c8fc98c42daaaeb2d280a72f594fc6f57b66e1ccf039fa"}
Sep 08 11:32:32.678794 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:32.678769    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:32.796723 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:32.796689    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:32.984527 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:32.984501    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:33.350297 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:33.350182    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:33.634606 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:33.634513    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwbt2" event={"ID":"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5","Type":"ContainerStarted","Data":"de488005d29caa6d4dae2191996e4a017f8d3cb3c490228ab63b3ed2bb2b9f1f"}
Sep 08 11:32:33.634606 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:33.634572    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwbt2" event={"ID":"8614060e-29d7-4d4a-b4a0-a3c52cc2caf5","Type":"ContainerStarted","Data":"8db3a36366dbf0de5e74c0eba8bd3222c0b1295b626ec89ea837bf52ac21a916"}
Sep 08 11:32:33.658307 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:33.658259    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kwbt2" podStartSLOduration=2.286557206 podStartE2EDuration="4.658243914s" totalImagesPullingTime="2.311404295s" totalInitContainerRuntime="60.282413ms" isStatefulPod=true podCreationTimestamp="2026-09-08 11:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:33.656371267 +0000 UTC m=+160.223131799" watchObservedRunningTime="2026-09-08 11:32:33.658243914 +0000 UTC m=+160.225004455"
Sep 08 11:32:33.954783 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:33.954752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:34.052443 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:34.052405    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:34.152049 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:34.152021    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:34.411558 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:34.411474    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:34.532765 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:34.532731    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:34.624726 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:34.624678    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:34.870678 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:34.870647    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:34.984880 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:34.984850    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:35.795208 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:35.795174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:32:35.798322 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:35.798288    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls\") pod \"image-registry-6c88c548cf-26prd\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") " pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:32:35.825424 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:35.825390    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-59gmm\""
Sep 08 11:32:35.833575 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:35.833532    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:32:35.896090 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:35.896051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:32:35.899067 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:35.899037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a77e471-aaa5-4f4d-932d-c8466b009a67-metrics-tls\") pod \"dns-default-67xmf\" (UID: \"1a77e471-aaa5-4f4d-932d-c8466b009a67\") " pod="openshift-dns/dns-default-67xmf"
Sep 08 11:32:35.960562 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:35.960522    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:36.113011 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:36.112985    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c88c548cf-26prd"]
Sep 08 11:32:36.115175 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:32:36.115113    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9d38ed_f77d_44c7_a9ed_31df4bf75d4d.slice/crio-a281bee0bf8e285780da3d829b717675241e2d9c12db54c8df54ee86d3819bdd WatchSource:0}: Error finding container a281bee0bf8e285780da3d829b717675241e2d9c12db54c8df54ee86d3819bdd: Status 404 returned error can't find the container with id a281bee0bf8e285780da3d829b717675241e2d9c12db54c8df54ee86d3819bdd
Sep 08 11:32:36.125257 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:36.125227    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2v99l\""
Sep 08 11:32:36.133357 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:36.133326    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-67xmf"
Sep 08 11:32:36.409128 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:36.409051    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-67xmf"]
Sep 08 11:32:36.412037 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:32:36.412006    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a77e471_aaa5_4f4d_932d_c8466b009a67.slice/crio-2888b850837f82cb8892b5a2bd583688c8ab4ca20bff72c317872bf07c98dd34 WatchSource:0}: Error finding container 2888b850837f82cb8892b5a2bd583688c8ab4ca20bff72c317872bf07c98dd34: Status 404 returned error can't find the container with id 2888b850837f82cb8892b5a2bd583688c8ab4ca20bff72c317872bf07c98dd34
Sep 08 11:32:36.527695 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:36.527662    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:36.627810 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:36.627778    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:36.645768 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:36.645735    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c88c548cf-26prd" event={"ID":"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d","Type":"ContainerStarted","Data":"c1f8110fac59c74fd9970725fe157df88c5c17aa3ba9e3e4df348caaf3f9bd11"}
Sep 08 11:32:36.645918 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:36.645786    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:32:36.645918 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:36.645807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c88c548cf-26prd" event={"ID":"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d","Type":"ContainerStarted","Data":"a281bee0bf8e285780da3d829b717675241e2d9c12db54c8df54ee86d3819bdd"}
Sep 08 11:32:36.646916 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:36.646897    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-67xmf" event={"ID":"1a77e471-aaa5-4f4d-932d-c8466b009a67","Type":"ContainerStarted","Data":"2888b850837f82cb8892b5a2bd583688c8ab4ca20bff72c317872bf07c98dd34"}
Sep 08 11:32:36.672827 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:36.672733    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c88c548cf-26prd" podStartSLOduration=162.672717533 podStartE2EDuration="2m42.672717533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:36.672221769 +0000 UTC m=+163.238982310" watchObservedRunningTime="2026-09-08 11:32:36.672717533 +0000 UTC m=+163.239478074"
Sep 08 11:32:38.656565 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:38.656448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-67xmf" event={"ID":"1a77e471-aaa5-4f4d-932d-c8466b009a67","Type":"ContainerStarted","Data":"3020f18faee2fdd9501a6beba7b68c71b22295427c06e90bcc8285c431629bc3"}
Sep 08 11:32:38.656565 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:38.656537    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-67xmf" event={"ID":"1a77e471-aaa5-4f4d-932d-c8466b009a67","Type":"ContainerStarted","Data":"f1e18bb93ab98dd680fee47dc044255ccbaa4549e1e6d889f8968f594024326f"}
Sep 08 11:32:38.656565 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:38.656565    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-67xmf"
Sep 08 11:32:38.675672 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:38.675626    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-67xmf" podStartSLOduration=129.707801113 podStartE2EDuration="2m11.67560906s" totalImagesPullingTime="1.967807947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:38.674736004 +0000 UTC m=+165.241496550" watchObservedRunningTime="2026-09-08 11:32:38.67560906 +0000 UTC m=+165.242369602"
Sep 08 11:32:39.211931 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.211895    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bcbd8d9f8-vwdcq"]
Sep 08 11:32:39.215011 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.214995    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.217533 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.217512    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 11:32:39.217656 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.217614    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 11:32:39.218593 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.218573    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 11:32:39.218699 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.218680    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 11:32:39.218817 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.218801    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 11:32:39.220070 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.220047    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 11:32:39.220169 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.220083    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 11:32:39.220169 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.220085    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4fm94\""
Sep 08 11:32:39.223664 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.223641    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 11:32:39.225938 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.225918    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bcbd8d9f8-vwdcq"]
Sep 08 11:32:39.322803 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.322771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-serving-cert\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.322803 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.322805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-trusted-ca-bundle\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.323045 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.322827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-oauth-config\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.323045 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.322846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-service-ca\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.323045 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.322928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-oauth-serving-cert\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.323045 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.322970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p4t9\" (UniqueName: \"kubernetes.io/projected/ccb913a7-eba5-4a23-b423-0759f5c5dc96-kube-api-access-8p4t9\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.323045 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.323031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-config\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.346581 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.346549    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c88c548cf-26prd"]
Sep 08 11:32:39.424180 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.424144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-trusted-ca-bundle\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.424180 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.424184    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-oauth-config\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.424460 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.424203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-service-ca\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.424460 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.424231    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-oauth-serving-cert\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.424460 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.424246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4t9\" (UniqueName: \"kubernetes.io/projected/ccb913a7-eba5-4a23-b423-0759f5c5dc96-kube-api-access-8p4t9\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.424460 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.424382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-config\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.424460 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.424455    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-serving-cert\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.425072 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.425044    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-oauth-serving-cert\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.425195 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.425093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-service-ca\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.425280 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.425258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-config\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.425412 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.425388    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-trusted-ca-bundle\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.426902 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.426878    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-serving-cert\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.426988 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.426888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-oauth-config\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.441522 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.441495    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4t9\" (UniqueName: \"kubernetes.io/projected/ccb913a7-eba5-4a23-b423-0759f5c5dc96-kube-api-access-8p4t9\") pod \"console-6bcbd8d9f8-vwdcq\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") " pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.444429 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.444403    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:39.524666 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.524569    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:39.548137 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.548110    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:39.795609 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:39.795579    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bcbd8d9f8-vwdcq"]
Sep 08 11:32:39.797867 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:32:39.797842    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb913a7_eba5_4a23_b423_0759f5c5dc96.slice/crio-2b548c27ff2f3f71c06a680702d4fcd34f761f0d92f04f4cdb418614eef1031a WatchSource:0}: Error finding container 2b548c27ff2f3f71c06a680702d4fcd34f761f0d92f04f4cdb418614eef1031a: Status 404 returned error can't find the container with id 2b548c27ff2f3f71c06a680702d4fcd34f761f0d92f04f4cdb418614eef1031a
Sep 08 11:32:40.664763 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:40.664728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcbd8d9f8-vwdcq" event={"ID":"ccb913a7-eba5-4a23-b423-0759f5c5dc96","Type":"ContainerStarted","Data":"2b548c27ff2f3f71c06a680702d4fcd34f761f0d92f04f4cdb418614eef1031a"}
Sep 08 11:32:43.678535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:43.678444    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcbd8d9f8-vwdcq" event={"ID":"ccb913a7-eba5-4a23-b423-0759f5c5dc96","Type":"ContainerStarted","Data":"56bb2c2afcb22e0ea79ae89f8035d06ec46107af1b2eb14099dc1ac3c238f4f7"}
Sep 08 11:32:43.699515 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:43.699450    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bcbd8d9f8-vwdcq" podStartSLOduration=1.24607488 podStartE2EDuration="4.699435088s" totalImagesPullingTime="3.453360208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:43.697443582 +0000 UTC m=+170.264204119" watchObservedRunningTime="2026-09-08 11:32:43.699435088 +0000 UTC m=+170.266195629"
Sep 08 11:32:47.178548 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:47.178454    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:47.279259 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:47.279225    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:48.662347 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:48.662314    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-67xmf"
Sep 08 11:32:49.524900 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:49.524861    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:49.525047 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:49.524928    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:49.530428 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:49.530403    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:49.704660 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:49.704626    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:32:49.884932 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:49.884855    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:49.984437 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:49.984404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:50.086671 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:50.086627    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:56.394322 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:56.394283    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:56.493871 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:56.493842    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:56.560669 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:56.560639    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:57.829177 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:57.829148    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:57.929941 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:57.929906    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:32:59.352607 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:32:59.352575    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:33:04.365476 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.365399    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c88c548cf-26prd" podUID="1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" containerName="registry" containerID="cri-o://c1f8110fac59c74fd9970725fe157df88c5c17aa3ba9e3e4df348caaf3f9bd11" gracePeriod=30
Sep 08 11:33:04.597627 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.597604    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:33:04.614368 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.614340    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-trusted-ca\") pod \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") "
Sep 08 11:33:04.614442 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.614399    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-ca-trust-extracted\") pod \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") "
Sep 08 11:33:04.614442 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.614420    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b795d\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-kube-api-access-b795d\") pod \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") "
Sep 08 11:33:04.614539 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.614454    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-image-registry-private-configuration\") pod \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") "
Sep 08 11:33:04.614539 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.614488    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-bound-sa-token\") pod \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") "
Sep 08 11:33:04.614641 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.614593    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-installation-pull-secrets\") pod \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") "
Sep 08 11:33:04.614641 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.614632    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls\") pod \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") "
Sep 08 11:33:04.614725 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.614672    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-certificates\") pod \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\" (UID: \"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d\") "
Sep 08 11:33:04.614803 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.614776    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:33:04.614946 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.614918    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-trusted-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:33:04.615229 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.615203    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:33:04.616872 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.616816    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:33:04.616977 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.616900    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:33:04.617049 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.617016    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-kube-api-access-b795d" (OuterVolumeSpecName: "kube-api-access-b795d") pod "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d"). InnerVolumeSpecName "kube-api-access-b795d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:33:04.617049 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.617024    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:33:04.617349 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.617282    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:33:04.624710 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.624662    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" (UID: "1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:33:04.709268 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.709233    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:33:04.715332 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.715311    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-image-registry-private-configuration\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:33:04.715393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.715345    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-bound-sa-token\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:33:04.715393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.715356    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-installation-pull-secrets\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:33:04.715393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.715365    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-tls\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:33:04.715393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.715373    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-registry-certificates\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:33:04.715393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.715381    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-ca-trust-extracted\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:33:04.715393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.715389    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b795d\" (UniqueName: \"kubernetes.io/projected/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d-kube-api-access-b795d\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:33:04.762821 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.762757    2561 generic.go:356] "Generic (PLEG): container finished" podID="1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" containerID="c1f8110fac59c74fd9970725fe157df88c5c17aa3ba9e3e4df348caaf3f9bd11" exitCode=0
Sep 08 11:33:04.763015 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.762865    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c88c548cf-26prd"
Sep 08 11:33:04.763015 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.762867    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c88c548cf-26prd" event={"ID":"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d","Type":"ContainerDied","Data":"c1f8110fac59c74fd9970725fe157df88c5c17aa3ba9e3e4df348caaf3f9bd11"}
Sep 08 11:33:04.763125 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.763019    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c88c548cf-26prd" event={"ID":"1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d","Type":"ContainerDied","Data":"a281bee0bf8e285780da3d829b717675241e2d9c12db54c8df54ee86d3819bdd"}
Sep 08 11:33:04.763125 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.763094    2561 scope.go:117] "RemoveContainer" containerID="c1f8110fac59c74fd9970725fe157df88c5c17aa3ba9e3e4df348caaf3f9bd11"
Sep 08 11:33:04.771514 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.771497    2561 scope.go:117] "RemoveContainer" containerID="c1f8110fac59c74fd9970725fe157df88c5c17aa3ba9e3e4df348caaf3f9bd11"
Sep 08 11:33:04.771774 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:33:04.771756    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f8110fac59c74fd9970725fe157df88c5c17aa3ba9e3e4df348caaf3f9bd11\": container with ID starting with c1f8110fac59c74fd9970725fe157df88c5c17aa3ba9e3e4df348caaf3f9bd11 not found: ID does not exist" containerID="c1f8110fac59c74fd9970725fe157df88c5c17aa3ba9e3e4df348caaf3f9bd11"
Sep 08 11:33:04.771825 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.771782    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f8110fac59c74fd9970725fe157df88c5c17aa3ba9e3e4df348caaf3f9bd11"} err="failed to get container status \"c1f8110fac59c74fd9970725fe157df88c5c17aa3ba9e3e4df348caaf3f9bd11\": rpc error: code = NotFound desc = could not find container \"c1f8110fac59c74fd9970725fe157df88c5c17aa3ba9e3e4df348caaf3f9bd11\": container with ID starting with c1f8110fac59c74fd9970725fe157df88c5c17aa3ba9e3e4df348caaf3f9bd11 not found: ID does not exist"
Sep 08 11:33:04.784877 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.784853    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c88c548cf-26prd"]
Sep 08 11:33:04.785455 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.785436    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:33:04.789544 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:04.789523    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c88c548cf-26prd"]
Sep 08 11:33:05.959802 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:05.959773    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" path="/var/lib/kubelet/pods/1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d/volumes"
Sep 08 11:33:19.624212 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:19.624174    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:33:19.747672 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:19.747636    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:33:19.836097 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:19.836058    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:33:22.432080 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:22.432030    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:33:45.761747 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:45.761714    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:33:45.841131 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:45.841097    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:33:57.720591 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:57.720552    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:33:57.789829 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:33:57.789800    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:34:04.706194 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:04.706158    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:34:04.808232 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:04.808206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:34:08.122241 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:08.122197    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:34:08.285286 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:08.285250    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:34:08.368089 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:08.368060    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:34:09.055032 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:09.054987    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:34:18.721645 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:18.721570    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bcbd8d9f8-vwdcq"]
Sep 08 11:34:18.846021 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:18.845960    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:34:18.945404 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:18.945374    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:34:19.024510 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:19.024424    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:34:43.740918 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:43.740874    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bcbd8d9f8-vwdcq" podUID="ccb913a7-eba5-4a23-b423-0759f5c5dc96" containerName="console" containerID="cri-o://56bb2c2afcb22e0ea79ae89f8035d06ec46107af1b2eb14099dc1ac3c238f4f7" gracePeriod=15
Sep 08 11:34:43.974379 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:43.974355    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:34:44.090523 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.090443    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-config\") pod \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") "
Sep 08 11:34:44.090523 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.090500    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-trusted-ca-bundle\") pod \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") "
Sep 08 11:34:44.090738 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.090578    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-oauth-serving-cert\") pod \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") "
Sep 08 11:34:44.090738 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.090624    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p4t9\" (UniqueName: \"kubernetes.io/projected/ccb913a7-eba5-4a23-b423-0759f5c5dc96-kube-api-access-8p4t9\") pod \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") "
Sep 08 11:34:44.090738 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.090649    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-serving-cert\") pod \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") "
Sep 08 11:34:44.090738 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.090694    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-service-ca\") pod \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") "
Sep 08 11:34:44.090738 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.090725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-oauth-config\") pod \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\" (UID: \"ccb913a7-eba5-4a23-b423-0759f5c5dc96\") "
Sep 08 11:34:44.090971 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.090865    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-config" (OuterVolumeSpecName: "console-config") pod "ccb913a7-eba5-4a23-b423-0759f5c5dc96" (UID: "ccb913a7-eba5-4a23-b423-0759f5c5dc96"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:34:44.090971 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.090877    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ccb913a7-eba5-4a23-b423-0759f5c5dc96" (UID: "ccb913a7-eba5-4a23-b423-0759f5c5dc96"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:34:44.090971 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.090915    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ccb913a7-eba5-4a23-b423-0759f5c5dc96" (UID: "ccb913a7-eba5-4a23-b423-0759f5c5dc96"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:34:44.091119 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.090969    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-config\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:34:44.091119 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.090988    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-trusted-ca-bundle\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:34:44.091301 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.091261    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-service-ca" (OuterVolumeSpecName: "service-ca") pod "ccb913a7-eba5-4a23-b423-0759f5c5dc96" (UID: "ccb913a7-eba5-4a23-b423-0759f5c5dc96"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:34:44.091827 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.091794    2561 generic.go:356] "Generic (PLEG): container finished" podID="ccb913a7-eba5-4a23-b423-0759f5c5dc96" containerID="56bb2c2afcb22e0ea79ae89f8035d06ec46107af1b2eb14099dc1ac3c238f4f7" exitCode=0
Sep 08 11:34:44.091944 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.091857    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcbd8d9f8-vwdcq"
Sep 08 11:34:44.091944 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.091872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcbd8d9f8-vwdcq" event={"ID":"ccb913a7-eba5-4a23-b423-0759f5c5dc96","Type":"ContainerDied","Data":"56bb2c2afcb22e0ea79ae89f8035d06ec46107af1b2eb14099dc1ac3c238f4f7"}
Sep 08 11:34:44.092054 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.091947    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcbd8d9f8-vwdcq" event={"ID":"ccb913a7-eba5-4a23-b423-0759f5c5dc96","Type":"ContainerDied","Data":"2b548c27ff2f3f71c06a680702d4fcd34f761f0d92f04f4cdb418614eef1031a"}
Sep 08 11:34:44.092054 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.091968    2561 scope.go:117] "RemoveContainer" containerID="56bb2c2afcb22e0ea79ae89f8035d06ec46107af1b2eb14099dc1ac3c238f4f7"
Sep 08 11:34:44.093251 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.093232    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ccb913a7-eba5-4a23-b423-0759f5c5dc96" (UID: "ccb913a7-eba5-4a23-b423-0759f5c5dc96"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:34:44.093313 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.093249    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb913a7-eba5-4a23-b423-0759f5c5dc96-kube-api-access-8p4t9" (OuterVolumeSpecName: "kube-api-access-8p4t9") pod "ccb913a7-eba5-4a23-b423-0759f5c5dc96" (UID: "ccb913a7-eba5-4a23-b423-0759f5c5dc96"). InnerVolumeSpecName "kube-api-access-8p4t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:34:44.093313 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.093253    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ccb913a7-eba5-4a23-b423-0759f5c5dc96" (UID: "ccb913a7-eba5-4a23-b423-0759f5c5dc96"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:34:44.105074 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.105057    2561 scope.go:117] "RemoveContainer" containerID="56bb2c2afcb22e0ea79ae89f8035d06ec46107af1b2eb14099dc1ac3c238f4f7"
Sep 08 11:34:44.105337 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:34:44.105321    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bb2c2afcb22e0ea79ae89f8035d06ec46107af1b2eb14099dc1ac3c238f4f7\": container with ID starting with 56bb2c2afcb22e0ea79ae89f8035d06ec46107af1b2eb14099dc1ac3c238f4f7 not found: ID does not exist" containerID="56bb2c2afcb22e0ea79ae89f8035d06ec46107af1b2eb14099dc1ac3c238f4f7"
Sep 08 11:34:44.105388 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.105344    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bb2c2afcb22e0ea79ae89f8035d06ec46107af1b2eb14099dc1ac3c238f4f7"} err="failed to get container status \"56bb2c2afcb22e0ea79ae89f8035d06ec46107af1b2eb14099dc1ac3c238f4f7\": rpc error: code = NotFound desc = could not find container \"56bb2c2afcb22e0ea79ae89f8035d06ec46107af1b2eb14099dc1ac3c238f4f7\": container with ID starting with 56bb2c2afcb22e0ea79ae89f8035d06ec46107af1b2eb14099dc1ac3c238f4f7 not found: ID does not exist"
Sep 08 11:34:44.191864 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.191840    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:34:44.191864 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.191862    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-oauth-config\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:34:44.191986 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.191872    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ccb913a7-eba5-4a23-b423-0759f5c5dc96-oauth-serving-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:34:44.191986 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.191880    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p4t9\" (UniqueName: \"kubernetes.io/projected/ccb913a7-eba5-4a23-b423-0759f5c5dc96-kube-api-access-8p4t9\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:34:44.191986 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.191890    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ccb913a7-eba5-4a23-b423-0759f5c5dc96-console-serving-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:34:44.415223 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.415196    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bcbd8d9f8-vwdcq"]
Sep 08 11:34:44.420893 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:44.420872    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bcbd8d9f8-vwdcq"]
Sep 08 11:34:45.959190 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:45.959157    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccb913a7-eba5-4a23-b423-0759f5c5dc96" path="/var/lib/kubelet/pods/ccb913a7-eba5-4a23-b423-0759f5c5dc96/volumes"
Sep 08 11:34:50.230127 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:50.230089    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:34:50.331830 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:50.331798    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:34:53.862304 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:34:53.862275    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 11:35:00.356940 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:00.356908    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:00.455021 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:00.454961    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:10.473644 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:10.473609    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:10.535434 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:10.535399    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:22.132926 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:22.132895    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:22.942180 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:22.942147    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:25.742696 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:25.742655    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:25.840639 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:25.840603    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:27.205342 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:27.205311    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:27.302517 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:27.302488    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:27.408971 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:27.408940    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:29.615615 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:29.615580    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:29.840305 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:29.840277    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:32.340822 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:32.340790    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:32.440323 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:32.440288    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:36.194857 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:36.194821    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:36.314230 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:36.314197    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:36.836879 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:36.836844    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:38.823058 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:38.823026    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:39.050069 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:39.050040    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:40.142510 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:40.142459    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:40.983138 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:40.983104    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24"]
Sep 08 11:35:40.983343 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:40.983331    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" containerName="registry"
Sep 08 11:35:40.983385 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:40.983344    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" containerName="registry"
Sep 08 11:35:40.983385 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:40.983357    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccb913a7-eba5-4a23-b423-0759f5c5dc96" containerName="console"
Sep 08 11:35:40.983385 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:40.983362    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccb913a7-eba5-4a23-b423-0759f5c5dc96" containerName="console"
Sep 08 11:35:40.983494 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:40.983403    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccb913a7-eba5-4a23-b423-0759f5c5dc96" containerName="console"
Sep 08 11:35:40.983494 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:40.983412    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b9d38ed-f77d-44c7-a9ed-31df4bf75d4d" containerName="registry"
Sep 08 11:35:40.985982 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:40.985966    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24"
Sep 08 11:35:40.990370 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:40.990338    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 11:35:40.990370 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:40.990352    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-lcp9s\""
Sep 08 11:35:40.990370 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:40.990339    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 11:35:41.007517 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:41.007492    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24"]
Sep 08 11:35:41.060615 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:41.060585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpcrm\" (UniqueName: \"kubernetes.io/projected/e72045e8-41b0-4bdd-a453-9a30cf92794f-kube-api-access-wpcrm\") pod \"openshift-lws-operator-fd8ccff4c-54s24\" (UID: \"e72045e8-41b0-4bdd-a453-9a30cf92794f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24"
Sep 08 11:35:41.060772 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:41.060622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e72045e8-41b0-4bdd-a453-9a30cf92794f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-54s24\" (UID: \"e72045e8-41b0-4bdd-a453-9a30cf92794f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24"
Sep 08 11:35:41.161678 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:41.161636    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpcrm\" (UniqueName: \"kubernetes.io/projected/e72045e8-41b0-4bdd-a453-9a30cf92794f-kube-api-access-wpcrm\") pod \"openshift-lws-operator-fd8ccff4c-54s24\" (UID: \"e72045e8-41b0-4bdd-a453-9a30cf92794f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24"
Sep 08 11:35:41.161678 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:41.161680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e72045e8-41b0-4bdd-a453-9a30cf92794f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-54s24\" (UID: \"e72045e8-41b0-4bdd-a453-9a30cf92794f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24"
Sep 08 11:35:41.162086 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:41.161997    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e72045e8-41b0-4bdd-a453-9a30cf92794f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-54s24\" (UID: \"e72045e8-41b0-4bdd-a453-9a30cf92794f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24"
Sep 08 11:35:41.171976 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:41.171953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpcrm\" (UniqueName: \"kubernetes.io/projected/e72045e8-41b0-4bdd-a453-9a30cf92794f-kube-api-access-wpcrm\") pod \"openshift-lws-operator-fd8ccff4c-54s24\" (UID: \"e72045e8-41b0-4bdd-a453-9a30cf92794f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24"
Sep 08 11:35:41.294619 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:41.294541    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24"
Sep 08 11:35:41.569786 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:41.569555    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24"]
Sep 08 11:35:41.572323 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:35:41.572284    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode72045e8_41b0_4bdd_a453_9a30cf92794f.slice/crio-0305aaa2274aacaa05caf8f7502c819696873bab439e226f9d30f47a8c8d3544 WatchSource:0}: Error finding container 0305aaa2274aacaa05caf8f7502c819696873bab439e226f9d30f47a8c8d3544: Status 404 returned error can't find the container with id 0305aaa2274aacaa05caf8f7502c819696873bab439e226f9d30f47a8c8d3544
Sep 08 11:35:41.573960 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:41.573946    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 11:35:42.283563 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:42.283532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24" event={"ID":"e72045e8-41b0-4bdd-a453-9a30cf92794f","Type":"ContainerStarted","Data":"0305aaa2274aacaa05caf8f7502c819696873bab439e226f9d30f47a8c8d3544"}
Sep 08 11:35:46.299623 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:46.299583    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24" event={"ID":"e72045e8-41b0-4bdd-a453-9a30cf92794f","Type":"ContainerStarted","Data":"40281ccdba589197a14c3f2e35882f482929361e88ec2d4210e70b89019e3295"}
Sep 08 11:35:46.321228 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:46.321181    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-54s24" podStartSLOduration=1.944939781 podStartE2EDuration="6.321167121s" totalImagesPullingTime="4.37622734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:35:46.319555598 +0000 UTC m=+352.886316139" watchObservedRunningTime="2026-09-08 11:35:46.321167121 +0000 UTC m=+352.887927662"
Sep 08 11:35:47.196312 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:47.196271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:47.291494 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:47.291427    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:35:49.202176 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:35:49.202139    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:01.174518 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:01.174457    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:01.254198 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:01.254164    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:01.705170 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:01.705138    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:01.916509 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:01.916461    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:02.055434 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.055354    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"]
Sep 08 11:36:02.058476 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.058451    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.061815 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.061797    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 11:36:02.062093 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.062074    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 11:36:02.062592 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.062578    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 11:36:02.062690 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.062673    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-646dc\""
Sep 08 11:36:02.062805 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.062792    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 11:36:02.080723 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.080695    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"]
Sep 08 11:36:02.097953 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.097923    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvxqz\" (UniqueName: \"kubernetes.io/projected/a746dbca-f2b7-47d3-b876-a9c476945d62-kube-api-access-wvxqz\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.098101 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.097961    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a746dbca-f2b7-47d3-b876-a9c476945d62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.098101 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.097990    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a746dbca-f2b7-47d3-b876-a9c476945d62-tmp\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.098101 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.098014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a746dbca-f2b7-47d3-b876-a9c476945d62-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.098101 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.098059    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a746dbca-f2b7-47d3-b876-a9c476945d62-manifests\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.104305 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.104280    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:02.198586 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.198544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a746dbca-f2b7-47d3-b876-a9c476945d62-manifests\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.198586 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.198591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxqz\" (UniqueName: \"kubernetes.io/projected/a746dbca-f2b7-47d3-b876-a9c476945d62-kube-api-access-wvxqz\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.199032 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.198720    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a746dbca-f2b7-47d3-b876-a9c476945d62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.199032 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.198769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a746dbca-f2b7-47d3-b876-a9c476945d62-tmp\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.199032 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.198808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a746dbca-f2b7-47d3-b876-a9c476945d62-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.199032 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.199021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a746dbca-f2b7-47d3-b876-a9c476945d62-manifests\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.201052 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.201021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a746dbca-f2b7-47d3-b876-a9c476945d62-tmp\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.201186 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.201155    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a746dbca-f2b7-47d3-b876-a9c476945d62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.201324 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.201304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a746dbca-f2b7-47d3-b876-a9c476945d62-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.212035 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.212007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxqz\" (UniqueName: \"kubernetes.io/projected/a746dbca-f2b7-47d3-b876-a9c476945d62-kube-api-access-wvxqz\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-x2l6s\" (UID: \"a746dbca-f2b7-47d3-b876-a9c476945d62\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.276320 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.276287    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:02.368223 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.368133    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:02.392618 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.392572    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:02.495684 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.495634    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:02.595599 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.595570    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:02.647391 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.647365    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"]
Sep 08 11:36:02.648921 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:36:02.648893    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda746dbca_f2b7_47d3_b876_a9c476945d62.slice/crio-01a1acc14936adbb6cdeb8c606d84fda0cbbe11f0ff3935e9c6464354393ebac WatchSource:0}: Error finding container 01a1acc14936adbb6cdeb8c606d84fda0cbbe11f0ff3935e9c6464354393ebac: Status 404 returned error can't find the container with id 01a1acc14936adbb6cdeb8c606d84fda0cbbe11f0ff3935e9c6464354393ebac
Sep 08 11:36:02.690584 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:02.690543    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:03.356662 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:03.356622    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s" event={"ID":"a746dbca-f2b7-47d3-b876-a9c476945d62","Type":"ContainerStarted","Data":"01a1acc14936adbb6cdeb8c606d84fda0cbbe11f0ff3935e9c6464354393ebac"}
Sep 08 11:36:06.377406 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:06.377373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s" event={"ID":"a746dbca-f2b7-47d3-b876-a9c476945d62","Type":"ContainerStarted","Data":"9bf47ef54e6619def64a18c72b36feb1a2a6912810ec2482fb77d92f91bd1a58"}
Sep 08 11:36:07.383585 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:07.383545    2561 generic.go:356] "Generic (PLEG): container finished" podID="a746dbca-f2b7-47d3-b876-a9c476945d62" containerID="9bf47ef54e6619def64a18c72b36feb1a2a6912810ec2482fb77d92f91bd1a58" exitCode=0
Sep 08 11:36:07.383996 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:07.383626    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s" event={"ID":"a746dbca-f2b7-47d3-b876-a9c476945d62","Type":"ContainerDied","Data":"9bf47ef54e6619def64a18c72b36feb1a2a6912810ec2482fb77d92f91bd1a58"}
Sep 08 11:36:07.441202 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:07.441168    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:07.531275 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:07.531240    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:08.388705 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:08.388670    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s" event={"ID":"a746dbca-f2b7-47d3-b876-a9c476945d62","Type":"ContainerStarted","Data":"8a2115cf47dd2af617f34f3ce4269087ae54d2e28053b8a9d43042cb0058c6c4"}
Sep 08 11:36:08.389095 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:08.388809    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:08.420244 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:08.420197    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s" podStartSLOduration=1.8142344929999998 podStartE2EDuration="6.420180885s" totalImagesPullingTime="3.577002449s" totalInitContainerRuntime="1.028943943s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:36:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:36:08.418545175 +0000 UTC m=+374.985305718" watchObservedRunningTime="2026-09-08 11:36:08.420180885 +0000 UTC m=+374.986941426"
Sep 08 11:36:19.394430 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:19.394395    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-x2l6s"
Sep 08 11:36:21.173022 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.172985    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:21.624873 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.624834    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"]
Sep 08 11:36:21.634655 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.634621    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"
Sep 08 11:36:21.637369 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.637344    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 11:36:21.637369 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.637350    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 11:36:21.638586 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.638545    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-l2f7h\""
Sep 08 11:36:21.638704 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.638553    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 11:36:21.638766 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.638557    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 11:36:21.640689 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.640655    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"]
Sep 08 11:36:21.735700 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.735656    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r55kr\" (UniqueName: \"kubernetes.io/projected/5f771995-14d3-4418-8046-34f682d39ceb-kube-api-access-r55kr\") pod \"kube-auth-proxy-7f8d66d765-tr8qv\" (UID: \"5f771995-14d3-4418-8046-34f682d39ceb\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"
Sep 08 11:36:21.735875 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.735708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f771995-14d3-4418-8046-34f682d39ceb-tls-certs\") pod \"kube-auth-proxy-7f8d66d765-tr8qv\" (UID: \"5f771995-14d3-4418-8046-34f682d39ceb\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"
Sep 08 11:36:21.735875 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.735805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f771995-14d3-4418-8046-34f682d39ceb-tmp\") pod \"kube-auth-proxy-7f8d66d765-tr8qv\" (UID: \"5f771995-14d3-4418-8046-34f682d39ceb\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"
Sep 08 11:36:21.836869 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.836824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r55kr\" (UniqueName: \"kubernetes.io/projected/5f771995-14d3-4418-8046-34f682d39ceb-kube-api-access-r55kr\") pod \"kube-auth-proxy-7f8d66d765-tr8qv\" (UID: \"5f771995-14d3-4418-8046-34f682d39ceb\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"
Sep 08 11:36:21.836869 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.836872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f771995-14d3-4418-8046-34f682d39ceb-tls-certs\") pod \"kube-auth-proxy-7f8d66d765-tr8qv\" (UID: \"5f771995-14d3-4418-8046-34f682d39ceb\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"
Sep 08 11:36:21.837118 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.836918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f771995-14d3-4418-8046-34f682d39ceb-tmp\") pod \"kube-auth-proxy-7f8d66d765-tr8qv\" (UID: \"5f771995-14d3-4418-8046-34f682d39ceb\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"
Sep 08 11:36:21.837118 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:36:21.837068    2561 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 08 11:36:21.837221 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:36:21.837165    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f771995-14d3-4418-8046-34f682d39ceb-tls-certs podName:5f771995-14d3-4418-8046-34f682d39ceb nodeName:}" failed. No retries permitted until 2026-09-08 11:36:22.337142322 +0000 UTC m=+388.903902864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5f771995-14d3-4418-8046-34f682d39ceb-tls-certs") pod "kube-auth-proxy-7f8d66d765-tr8qv" (UID: "5f771995-14d3-4418-8046-34f682d39ceb") : secret "kube-auth-proxy-tls" not found
Sep 08 11:36:21.839216 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.839190    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f771995-14d3-4418-8046-34f682d39ceb-tmp\") pod \"kube-auth-proxy-7f8d66d765-tr8qv\" (UID: \"5f771995-14d3-4418-8046-34f682d39ceb\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"
Sep 08 11:36:21.848395 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:21.848358    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r55kr\" (UniqueName: \"kubernetes.io/projected/5f771995-14d3-4418-8046-34f682d39ceb-kube-api-access-r55kr\") pod \"kube-auth-proxy-7f8d66d765-tr8qv\" (UID: \"5f771995-14d3-4418-8046-34f682d39ceb\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"
Sep 08 11:36:22.014388 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.014348    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"]
Sep 08 11:36:22.017534 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.017514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.021642 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.021620    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rlc4f\""
Sep 08 11:36:22.022492 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.022456    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 11:36:22.022739 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.022720    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 11:36:22.022996 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.022980    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 11:36:22.034832 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.034806    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"]
Sep 08 11:36:22.038889 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.038863    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2-metrics-cert\") pod \"lws-controller-manager-b949f5b5c-wtpbw\" (UID: \"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2\") " pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.039002 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.038940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2-manager-config\") pod \"lws-controller-manager-b949f5b5c-wtpbw\" (UID: \"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2\") " pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.039002 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.038973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2-cert\") pod \"lws-controller-manager-b949f5b5c-wtpbw\" (UID: \"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2\") " pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.039073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.039053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmsl\" (UniqueName: \"kubernetes.io/projected/852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2-kube-api-access-qlmsl\") pod \"lws-controller-manager-b949f5b5c-wtpbw\" (UID: \"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2\") " pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.140240 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.140201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2-cert\") pod \"lws-controller-manager-b949f5b5c-wtpbw\" (UID: \"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2\") " pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.140415 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.140253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmsl\" (UniqueName: \"kubernetes.io/projected/852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2-kube-api-access-qlmsl\") pod \"lws-controller-manager-b949f5b5c-wtpbw\" (UID: \"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2\") " pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.140415 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.140285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2-metrics-cert\") pod \"lws-controller-manager-b949f5b5c-wtpbw\" (UID: \"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2\") " pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.140415 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.140324    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2-manager-config\") pod \"lws-controller-manager-b949f5b5c-wtpbw\" (UID: \"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2\") " pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.141000 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.140975    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2-manager-config\") pod \"lws-controller-manager-b949f5b5c-wtpbw\" (UID: \"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2\") " pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.142689 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.142660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2-metrics-cert\") pod \"lws-controller-manager-b949f5b5c-wtpbw\" (UID: \"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2\") " pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.142778 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.142739    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2-cert\") pod \"lws-controller-manager-b949f5b5c-wtpbw\" (UID: \"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2\") " pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.149549 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.149528    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmsl\" (UniqueName: \"kubernetes.io/projected/852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2-kube-api-access-qlmsl\") pod \"lws-controller-manager-b949f5b5c-wtpbw\" (UID: \"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2\") " pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.326762 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.326666    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:22.342363 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.342323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f771995-14d3-4418-8046-34f682d39ceb-tls-certs\") pod \"kube-auth-proxy-7f8d66d765-tr8qv\" (UID: \"5f771995-14d3-4418-8046-34f682d39ceb\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"
Sep 08 11:36:22.345181 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.345158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f771995-14d3-4418-8046-34f682d39ceb-tls-certs\") pod \"kube-auth-proxy-7f8d66d765-tr8qv\" (UID: \"5f771995-14d3-4418-8046-34f682d39ceb\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"
Sep 08 11:36:22.546232 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.546191    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"
Sep 08 11:36:22.603557 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.603034    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"]
Sep 08 11:36:22.607286 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:36:22.607256    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852e9af0_b0b4_40f1_b4ca_a8a39ae5d2b2.slice/crio-1394c08673b2178914a83b1aa950a8b62cf5ef0a670c4e8f3833b9b1dd44002f WatchSource:0}: Error finding container 1394c08673b2178914a83b1aa950a8b62cf5ef0a670c4e8f3833b9b1dd44002f: Status 404 returned error can't find the container with id 1394c08673b2178914a83b1aa950a8b62cf5ef0a670c4e8f3833b9b1dd44002f
Sep 08 11:36:22.790479 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.790441    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:22.820820 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.820779    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv"]
Sep 08 11:36:22.823621 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:36:22.823591    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f771995_14d3_4418_8046_34f682d39ceb.slice/crio-f7c869f7d524a420c84413beb3d72532fdf296eff3aecf8df8a6ba2d5321cbf8 WatchSource:0}: Error finding container f7c869f7d524a420c84413beb3d72532fdf296eff3aecf8df8a6ba2d5321cbf8: Status 404 returned error can't find the container with id f7c869f7d524a420c84413beb3d72532fdf296eff3aecf8df8a6ba2d5321cbf8
Sep 08 11:36:22.897985 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:22.897885    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:23.351697 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:23.351657    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:23.447644 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:23.447607    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw" event={"ID":"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2","Type":"ContainerStarted","Data":"1394c08673b2178914a83b1aa950a8b62cf5ef0a670c4e8f3833b9b1dd44002f"}
Sep 08 11:36:23.448687 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:23.448668    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv" event={"ID":"5f771995-14d3-4418-8046-34f682d39ceb","Type":"ContainerStarted","Data":"f7c869f7d524a420c84413beb3d72532fdf296eff3aecf8df8a6ba2d5321cbf8"}
Sep 08 11:36:23.452446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:23.452426    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:23.854638 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:23.854601    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:25.299870 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:25.299830    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:26.462706 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:26.462673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw" event={"ID":"852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2","Type":"ContainerStarted","Data":"bcdd5412a70c102042f341611ec628ce677a28151ec7151ff51a72d0166930ee"}
Sep 08 11:36:26.463091 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:26.462798    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:26.464214 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:26.464190    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv" event={"ID":"5f771995-14d3-4418-8046-34f682d39ceb","Type":"ContainerStarted","Data":"f6c486c451c3967db043f5b7a6831d296202adf1e00ebb44f1b3b3f9bdfdcdd4"}
Sep 08 11:36:26.487913 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:26.487869    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw" podStartSLOduration=1.987594914 podStartE2EDuration="5.487854554s" totalImagesPullingTime="3.50025964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:36:26.486602833 +0000 UTC m=+393.053363374" watchObservedRunningTime="2026-09-08 11:36:26.487854554 +0000 UTC m=+393.054615095"
Sep 08 11:36:26.523901 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:26.523848    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f8d66d765-tr8qv" podStartSLOduration=2.19038965 podStartE2EDuration="5.523832432s" totalImagesPullingTime="3.333442782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:36:26.52256979 +0000 UTC m=+393.089330329" watchObservedRunningTime="2026-09-08 11:36:26.523832432 +0000 UTC m=+393.090592972"
Sep 08 11:36:28.365305 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:28.365259    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:28.465993 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:28.465964    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:29.075241 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:29.075206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:29.230310 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:29.230275    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:29.408441 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:29.408354    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:32.456602 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:32.456568    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:32.556534 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:32.556493    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:32.657695 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:32.657664    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:32.844818 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:32.844738    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:33.066402 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:33.066373    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:33.538290 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:33.538254    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:37.470932 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:37.470902    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-b949f5b5c-wtpbw"
Sep 08 11:36:42.066195 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:42.066158    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:42.158272 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:42.158209    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:42.340248 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:42.340154    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:42.440811 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:42.440782    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:43.245718 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:43.245671    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:43.349263 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:43.349228    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:36:43.457705 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:36:43.457674    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:21.363392 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:21.363262    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:21.464616 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:21.464552    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:29.126874 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:29.126833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:32.746892 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:32.746840    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:32.849387 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:32.849354    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:32.925523 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:32.925462    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:33.448552 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:33.448513    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:33.544348 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:33.544320    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:46.123683 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:46.123644    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:47.105312 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:47.105256    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:47.926808 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:47.926771    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:49.957836 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:49.957786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:50.058288 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:50.058240    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:51.339576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:51.339539    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:53.168788 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:53.168736    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:53.328434 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:53.328392    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:55.122116 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:55.122084    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:55.351524 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:55.351462    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:55.536620 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:55.536585    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:55.657207 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:55.657166    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:37:56.757785 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:37:56.757748    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:01.166570 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:01.166540    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:01.266447 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:01.266405    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:01.533678 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:01.533636    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:01.634419 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:01.634386    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:01.925540 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:01.925421    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:01.988387 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:01.988351    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:02.505456 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.505427    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:02.505902 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.505880    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"]
Sep 08 11:38:02.508862 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.508845    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"
Sep 08 11:38:02.511556 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.511533    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 11:38:02.511674 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.511533    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 11:38:02.512591 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.512567    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 08 11:38:02.512591 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.512588    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 08 11:38:02.512782 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.512593    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vjdc9\""
Sep 08 11:38:02.519481 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.519445    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"]
Sep 08 11:38:02.635125 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.635085    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5cl\" (UniqueName: \"kubernetes.io/projected/ce05a73a-33fa-4b6f-9d13-95a960e2f0a4-kube-api-access-lj5cl\") pod \"kuadrant-console-plugin-598f86468b-rwc5p\" (UID: \"ce05a73a-33fa-4b6f-9d13-95a960e2f0a4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"
Sep 08 11:38:02.635336 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.635145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ce05a73a-33fa-4b6f-9d13-95a960e2f0a4-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-rwc5p\" (UID: \"ce05a73a-33fa-4b6f-9d13-95a960e2f0a4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"
Sep 08 11:38:02.635336 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.635264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce05a73a-33fa-4b6f-9d13-95a960e2f0a4-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rwc5p\" (UID: \"ce05a73a-33fa-4b6f-9d13-95a960e2f0a4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"
Sep 08 11:38:02.736420 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.736379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce05a73a-33fa-4b6f-9d13-95a960e2f0a4-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rwc5p\" (UID: \"ce05a73a-33fa-4b6f-9d13-95a960e2f0a4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"
Sep 08 11:38:02.736623 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.736441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5cl\" (UniqueName: \"kubernetes.io/projected/ce05a73a-33fa-4b6f-9d13-95a960e2f0a4-kube-api-access-lj5cl\") pod \"kuadrant-console-plugin-598f86468b-rwc5p\" (UID: \"ce05a73a-33fa-4b6f-9d13-95a960e2f0a4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"
Sep 08 11:38:02.736623 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.736480    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ce05a73a-33fa-4b6f-9d13-95a960e2f0a4-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-rwc5p\" (UID: \"ce05a73a-33fa-4b6f-9d13-95a960e2f0a4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"
Sep 08 11:38:02.736623 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:38:02.736564    2561 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 08 11:38:02.736785 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:38:02.736652    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce05a73a-33fa-4b6f-9d13-95a960e2f0a4-plugin-serving-cert podName:ce05a73a-33fa-4b6f-9d13-95a960e2f0a4 nodeName:}" failed. No retries permitted until 2026-09-08 11:38:03.236632447 +0000 UTC m=+489.803392988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/ce05a73a-33fa-4b6f-9d13-95a960e2f0a4-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-rwc5p" (UID: "ce05a73a-33fa-4b6f-9d13-95a960e2f0a4") : secret "plugin-serving-cert" not found
Sep 08 11:38:02.737134 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.737115    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ce05a73a-33fa-4b6f-9d13-95a960e2f0a4-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-rwc5p\" (UID: \"ce05a73a-33fa-4b6f-9d13-95a960e2f0a4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"
Sep 08 11:38:02.746755 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:02.746719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5cl\" (UniqueName: \"kubernetes.io/projected/ce05a73a-33fa-4b6f-9d13-95a960e2f0a4-kube-api-access-lj5cl\") pod \"kuadrant-console-plugin-598f86468b-rwc5p\" (UID: \"ce05a73a-33fa-4b6f-9d13-95a960e2f0a4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"
Sep 08 11:38:03.170884 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:03.170848    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:03.240388 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:03.240351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce05a73a-33fa-4b6f-9d13-95a960e2f0a4-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rwc5p\" (UID: \"ce05a73a-33fa-4b6f-9d13-95a960e2f0a4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"
Sep 08 11:38:03.242787 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:03.242765    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce05a73a-33fa-4b6f-9d13-95a960e2f0a4-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-rwc5p\" (UID: \"ce05a73a-33fa-4b6f-9d13-95a960e2f0a4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"
Sep 08 11:38:03.418494 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:03.418429    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"
Sep 08 11:38:03.689145 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:03.688984    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p"]
Sep 08 11:38:03.691732 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:38:03.691704    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce05a73a_33fa_4b6f_9d13_95a960e2f0a4.slice/crio-2cfff7b5702fe6a3807ec04fe156f0a8a2c315a8d2dc0b2cd19aa2b72a4c41d8 WatchSource:0}: Error finding container 2cfff7b5702fe6a3807ec04fe156f0a8a2c315a8d2dc0b2cd19aa2b72a4c41d8: Status 404 returned error can't find the container with id 2cfff7b5702fe6a3807ec04fe156f0a8a2c315a8d2dc0b2cd19aa2b72a4c41d8
Sep 08 11:38:03.836897 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:03.836861    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p" event={"ID":"ce05a73a-33fa-4b6f-9d13-95a960e2f0a4","Type":"ContainerStarted","Data":"2cfff7b5702fe6a3807ec04fe156f0a8a2c315a8d2dc0b2cd19aa2b72a4c41d8"}
Sep 08 11:38:07.861740 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:07.861701    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:07.962821 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:07.962784    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:08.154530 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:08.154438    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:08.331777 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:08.331716    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:10.602845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:10.602814    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:10.671762 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:10.671734    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:17.155720 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:17.155680    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:17.255726 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:17.255687    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:19.212743 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:19.212615    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:19.348343 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:19.348304    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:22.344685 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:22.344647    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:22.447943 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:22.447911    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:22.752785 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:22.752752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:22.854644 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:22.854615    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:22.909669 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:22.909625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p" event={"ID":"ce05a73a-33fa-4b6f-9d13-95a960e2f0a4","Type":"ContainerStarted","Data":"a6038b2b1280d77d4e454cc61333ab73f662a0f178c13242e3fc80b32bed03b0"}
Sep 08 11:38:22.938138 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:22.938082    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-rwc5p" podStartSLOduration=1.806341612 podStartE2EDuration="20.938065669s" totalImagesPullingTime="19.131724057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:38:22.936215418 +0000 UTC m=+509.502975960" watchObservedRunningTime="2026-09-08 11:38:22.938065669 +0000 UTC m=+509.504826210"
Sep 08 11:38:22.964632 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:22.964598    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:23.030036 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:23.029921    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:29.971254 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:29.971213    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:30.077672 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:30.077629    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:32.424779 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:32.424744    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:32.525199 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:32.525149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:38.977849 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:38.977811    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:39.075194 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:39.075158    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:39.924732 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:39.924700    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:47.383078 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:47.382975    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:47.596682 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:47.596650    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:50.154604 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:50.154551    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:50.256764 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:50.256723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:50.360136 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:50.360100    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:50.999553 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:50.999518    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-hdxp8"]
Sep 08 11:38:51.002681 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.002662    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-hdxp8"
Sep 08 11:38:51.005858 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.005832    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lksmw\""
Sep 08 11:38:51.010592 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.010569    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-hdxp8"]
Sep 08 11:38:51.045692 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.045656    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:51.103481 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.103439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqg7n\" (UniqueName: \"kubernetes.io/projected/be35485f-88e6-47d2-8d00-5a7aca0cb4b3-kube-api-access-mqg7n\") pod \"authorino-b54df94b4-hdxp8\" (UID: \"be35485f-88e6-47d2-8d00-5a7aca0cb4b3\") " pod="kuadrant-system/authorino-b54df94b4-hdxp8"
Sep 08 11:38:51.130752 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.130721    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:38:51.141717 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.141680    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-p9d4z"]
Sep 08 11:38:51.144927 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.144904    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-p9d4z"
Sep 08 11:38:51.158871 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.158844    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-p9d4z"]
Sep 08 11:38:51.204426 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.204382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqg7n\" (UniqueName: \"kubernetes.io/projected/be35485f-88e6-47d2-8d00-5a7aca0cb4b3-kube-api-access-mqg7n\") pod \"authorino-b54df94b4-hdxp8\" (UID: \"be35485f-88e6-47d2-8d00-5a7aca0cb4b3\") " pod="kuadrant-system/authorino-b54df94b4-hdxp8"
Sep 08 11:38:51.204613 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.204443    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvnt5\" (UniqueName: \"kubernetes.io/projected/caf7bdb7-ef63-41fb-8c93-87ca879864e7-kube-api-access-hvnt5\") pod \"authorino-c548b5d7d-p9d4z\" (UID: \"caf7bdb7-ef63-41fb-8c93-87ca879864e7\") " pod="kuadrant-system/authorino-c548b5d7d-p9d4z"
Sep 08 11:38:51.216016 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.215990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqg7n\" (UniqueName: \"kubernetes.io/projected/be35485f-88e6-47d2-8d00-5a7aca0cb4b3-kube-api-access-mqg7n\") pod \"authorino-b54df94b4-hdxp8\" (UID: \"be35485f-88e6-47d2-8d00-5a7aca0cb4b3\") " pod="kuadrant-system/authorino-b54df94b4-hdxp8"
Sep 08 11:38:51.305072 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.304973    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvnt5\" (UniqueName: \"kubernetes.io/projected/caf7bdb7-ef63-41fb-8c93-87ca879864e7-kube-api-access-hvnt5\") pod \"authorino-c548b5d7d-p9d4z\" (UID: \"caf7bdb7-ef63-41fb-8c93-87ca879864e7\") " pod="kuadrant-system/authorino-c548b5d7d-p9d4z"
Sep 08 11:38:51.311896 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.311865    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-hdxp8"
Sep 08 11:38:51.316758 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.316737    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvnt5\" (UniqueName: \"kubernetes.io/projected/caf7bdb7-ef63-41fb-8c93-87ca879864e7-kube-api-access-hvnt5\") pod \"authorino-c548b5d7d-p9d4z\" (UID: \"caf7bdb7-ef63-41fb-8c93-87ca879864e7\") " pod="kuadrant-system/authorino-c548b5d7d-p9d4z"
Sep 08 11:38:51.453798 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.453745    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-p9d4z"
Sep 08 11:38:51.580888 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.580646    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-hdxp8"]
Sep 08 11:38:51.583097 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:38:51.583057    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe35485f_88e6_47d2_8d00_5a7aca0cb4b3.slice/crio-91b50aab4a9891eeaad5eaa5a912d4858ffcb6e294a2555aefca5e882956d7cc WatchSource:0}: Error finding container 91b50aab4a9891eeaad5eaa5a912d4858ffcb6e294a2555aefca5e882956d7cc: Status 404 returned error can't find the container with id 91b50aab4a9891eeaad5eaa5a912d4858ffcb6e294a2555aefca5e882956d7cc
Sep 08 11:38:51.725212 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:51.725118    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-p9d4z"]
Sep 08 11:38:51.727721 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:38:51.727678    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaf7bdb7_ef63_41fb_8c93_87ca879864e7.slice/crio-9dc3b7d61b626e781625401345a64e1f91e03c437eb343d7ba0b208af58e72ca WatchSource:0}: Error finding container 9dc3b7d61b626e781625401345a64e1f91e03c437eb343d7ba0b208af58e72ca: Status 404 returned error can't find the container with id 9dc3b7d61b626e781625401345a64e1f91e03c437eb343d7ba0b208af58e72ca
Sep 08 11:38:52.022635 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:52.022595    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-hdxp8" event={"ID":"be35485f-88e6-47d2-8d00-5a7aca0cb4b3","Type":"ContainerStarted","Data":"91b50aab4a9891eeaad5eaa5a912d4858ffcb6e294a2555aefca5e882956d7cc"}
Sep 08 11:38:52.023970 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:52.023940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-p9d4z" event={"ID":"caf7bdb7-ef63-41fb-8c93-87ca879864e7","Type":"ContainerStarted","Data":"9dc3b7d61b626e781625401345a64e1f91e03c437eb343d7ba0b208af58e72ca"}
Sep 08 11:38:56.044139 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:56.044104    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-hdxp8" event={"ID":"be35485f-88e6-47d2-8d00-5a7aca0cb4b3","Type":"ContainerStarted","Data":"de5d69f2a94f32e737d197cc0149d8f66862a308a7e940084f2e41dc035c2dd4"}
Sep 08 11:38:56.045721 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:56.045695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-p9d4z" event={"ID":"caf7bdb7-ef63-41fb-8c93-87ca879864e7","Type":"ContainerStarted","Data":"0a52d74d0d06cd06eea8d956d4aaf49a06de17a2ada7e48256288a568904a12e"}
Sep 08 11:38:56.073670 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:56.073607    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-hdxp8" podStartSLOduration=1.956881429 podStartE2EDuration="6.073590126s" totalImagesPullingTime="4.116708697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:38:56.072597323 +0000 UTC m=+542.639357865" watchObservedRunningTime="2026-09-08 11:38:56.073590126 +0000 UTC m=+542.640350668"
Sep 08 11:38:56.102508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:56.102359    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-p9d4z" podStartSLOduration=1.121617643 podStartE2EDuration="5.102340562s" totalImagesPullingTime="3.980722919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:38:56.102306964 +0000 UTC m=+542.669067505" watchObservedRunningTime="2026-09-08 11:38:56.102340562 +0000 UTC m=+542.669101106"
Sep 08 11:38:56.134845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:56.134808    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-hdxp8"]
Sep 08 11:38:58.054219 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:58.054176    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-hdxp8" podUID="be35485f-88e6-47d2-8d00-5a7aca0cb4b3" containerName="authorino" containerID="cri-o://de5d69f2a94f32e737d197cc0149d8f66862a308a7e940084f2e41dc035c2dd4" gracePeriod=30
Sep 08 11:38:58.296827 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:58.296800    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-hdxp8"
Sep 08 11:38:58.362004 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:58.361904    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqg7n\" (UniqueName: \"kubernetes.io/projected/be35485f-88e6-47d2-8d00-5a7aca0cb4b3-kube-api-access-mqg7n\") pod \"be35485f-88e6-47d2-8d00-5a7aca0cb4b3\" (UID: \"be35485f-88e6-47d2-8d00-5a7aca0cb4b3\") "
Sep 08 11:38:58.364101 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:58.364065    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be35485f-88e6-47d2-8d00-5a7aca0cb4b3-kube-api-access-mqg7n" (OuterVolumeSpecName: "kube-api-access-mqg7n") pod "be35485f-88e6-47d2-8d00-5a7aca0cb4b3" (UID: "be35485f-88e6-47d2-8d00-5a7aca0cb4b3"). InnerVolumeSpecName "kube-api-access-mqg7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:38:58.462654 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:58.462598    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqg7n\" (UniqueName: \"kubernetes.io/projected/be35485f-88e6-47d2-8d00-5a7aca0cb4b3-kube-api-access-mqg7n\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:38:59.059876 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:59.059841    2561 generic.go:356] "Generic (PLEG): container finished" podID="be35485f-88e6-47d2-8d00-5a7aca0cb4b3" containerID="de5d69f2a94f32e737d197cc0149d8f66862a308a7e940084f2e41dc035c2dd4" exitCode=0
Sep 08 11:38:59.060317 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:59.059884    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-hdxp8"
Sep 08 11:38:59.060317 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:59.059924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-hdxp8" event={"ID":"be35485f-88e6-47d2-8d00-5a7aca0cb4b3","Type":"ContainerDied","Data":"de5d69f2a94f32e737d197cc0149d8f66862a308a7e940084f2e41dc035c2dd4"}
Sep 08 11:38:59.060317 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:59.059964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-hdxp8" event={"ID":"be35485f-88e6-47d2-8d00-5a7aca0cb4b3","Type":"ContainerDied","Data":"91b50aab4a9891eeaad5eaa5a912d4858ffcb6e294a2555aefca5e882956d7cc"}
Sep 08 11:38:59.060317 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:59.059981    2561 scope.go:117] "RemoveContainer" containerID="de5d69f2a94f32e737d197cc0149d8f66862a308a7e940084f2e41dc035c2dd4"
Sep 08 11:38:59.068421 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:59.068405    2561 scope.go:117] "RemoveContainer" containerID="de5d69f2a94f32e737d197cc0149d8f66862a308a7e940084f2e41dc035c2dd4"
Sep 08 11:38:59.068728 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:38:59.068708    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5d69f2a94f32e737d197cc0149d8f66862a308a7e940084f2e41dc035c2dd4\": container with ID starting with de5d69f2a94f32e737d197cc0149d8f66862a308a7e940084f2e41dc035c2dd4 not found: ID does not exist" containerID="de5d69f2a94f32e737d197cc0149d8f66862a308a7e940084f2e41dc035c2dd4"
Sep 08 11:38:59.068789 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:59.068737    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5d69f2a94f32e737d197cc0149d8f66862a308a7e940084f2e41dc035c2dd4"} err="failed to get container status \"de5d69f2a94f32e737d197cc0149d8f66862a308a7e940084f2e41dc035c2dd4\": rpc error: code = NotFound desc = could not find container \"de5d69f2a94f32e737d197cc0149d8f66862a308a7e940084f2e41dc035c2dd4\": container with ID starting with de5d69f2a94f32e737d197cc0149d8f66862a308a7e940084f2e41dc035c2dd4 not found: ID does not exist"
Sep 08 11:38:59.081980 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:59.081956    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-hdxp8"]
Sep 08 11:38:59.086554 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:59.086533    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-hdxp8"]
Sep 08 11:38:59.959566 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:38:59.959536    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be35485f-88e6-47d2-8d00-5a7aca0cb4b3" path="/var/lib/kubelet/pods/be35485f-88e6-47d2-8d00-5a7aca0cb4b3/volumes"
Sep 08 11:39:10.385324 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:10.385290    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:39:10.485795 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:10.485760    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:39:10.889610 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:10.889570    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:39:35.655766 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:35.655730    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:39:35.900818 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:35.900782    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-swl9t"]
Sep 08 11:39:35.901100 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:35.901088    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be35485f-88e6-47d2-8d00-5a7aca0cb4b3" containerName="authorino"
Sep 08 11:39:35.901142 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:35.901102    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="be35485f-88e6-47d2-8d00-5a7aca0cb4b3" containerName="authorino"
Sep 08 11:39:35.901176 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:35.901149    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="be35485f-88e6-47d2-8d00-5a7aca0cb4b3" containerName="authorino"
Sep 08 11:39:35.906665 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:35.906618    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-swl9t"
Sep 08 11:39:35.916230 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:35.916205    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-swl9t"]
Sep 08 11:39:35.977317 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:35.977286    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:39:36.046194 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.046153    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-swl9t"]
Sep 08 11:39:36.046441 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:39:36.046415    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mvqkr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-swl9t" podUID="10a17813-dcee-42e7-a38c-ab434db3fbdc"
Sep 08 11:39:36.073223 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.073193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvqkr\" (UniqueName: \"kubernetes.io/projected/10a17813-dcee-42e7-a38c-ab434db3fbdc-kube-api-access-mvqkr\") pod \"authorino-5dc7ccb5f-swl9t\" (UID: \"10a17813-dcee-42e7-a38c-ab434db3fbdc\") " pod="kuadrant-system/authorino-5dc7ccb5f-swl9t"
Sep 08 11:39:36.076938 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.076915    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-2jpsk"]
Sep 08 11:39:36.079937 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.079921    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-2jpsk"
Sep 08 11:39:36.082614 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.082596    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 08 11:39:36.092562 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.092537    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-2jpsk"]
Sep 08 11:39:36.156439 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.156401    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-2jpsk"]
Sep 08 11:39:36.156840 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:39:36.156776    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tjvvz tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-2jpsk" podUID="444ed13f-4e47-477d-a751-e1c9c20e0e7c"
Sep 08 11:39:36.173796 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.173766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/444ed13f-4e47-477d-a751-e1c9c20e0e7c-tls-cert\") pod \"authorino-687bd45585-2jpsk\" (UID: \"444ed13f-4e47-477d-a751-e1c9c20e0e7c\") " pod="kuadrant-system/authorino-687bd45585-2jpsk"
Sep 08 11:39:36.173878 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.173807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvvz\" (UniqueName: \"kubernetes.io/projected/444ed13f-4e47-477d-a751-e1c9c20e0e7c-kube-api-access-tjvvz\") pod \"authorino-687bd45585-2jpsk\" (UID: \"444ed13f-4e47-477d-a751-e1c9c20e0e7c\") " pod="kuadrant-system/authorino-687bd45585-2jpsk"
Sep 08 11:39:36.173916 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.173878    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqkr\" (UniqueName: \"kubernetes.io/projected/10a17813-dcee-42e7-a38c-ab434db3fbdc-kube-api-access-mvqkr\") pod \"authorino-5dc7ccb5f-swl9t\" (UID: \"10a17813-dcee-42e7-a38c-ab434db3fbdc\") " pod="kuadrant-system/authorino-5dc7ccb5f-swl9t"
Sep 08 11:39:36.188896 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.188863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqkr\" (UniqueName: \"kubernetes.io/projected/10a17813-dcee-42e7-a38c-ab434db3fbdc-kube-api-access-mvqkr\") pod \"authorino-5dc7ccb5f-swl9t\" (UID: \"10a17813-dcee-42e7-a38c-ab434db3fbdc\") " pod="kuadrant-system/authorino-5dc7ccb5f-swl9t"
Sep 08 11:39:36.196770 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.196748    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bfd87fdb4-6t9w5"]
Sep 08 11:39:36.199788 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.199772    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5"
Sep 08 11:39:36.207818 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.207794    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-swl9t"
Sep 08 11:39:36.207818 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.207811    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-2jpsk"
Sep 08 11:39:36.210619 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.210599    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bfd87fdb4-6t9w5"]
Sep 08 11:39:36.214605 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.214585    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-swl9t"
Sep 08 11:39:36.217798 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.217781    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-2jpsk"
Sep 08 11:39:36.274663 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.274631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/444ed13f-4e47-477d-a751-e1c9c20e0e7c-tls-cert\") pod \"authorino-687bd45585-2jpsk\" (UID: \"444ed13f-4e47-477d-a751-e1c9c20e0e7c\") " pod="kuadrant-system/authorino-687bd45585-2jpsk"
Sep 08 11:39:36.274829 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.274674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvvz\" (UniqueName: \"kubernetes.io/projected/444ed13f-4e47-477d-a751-e1c9c20e0e7c-kube-api-access-tjvvz\") pod \"authorino-687bd45585-2jpsk\" (UID: \"444ed13f-4e47-477d-a751-e1c9c20e0e7c\") " pod="kuadrant-system/authorino-687bd45585-2jpsk"
Sep 08 11:39:36.276985 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.276961    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/444ed13f-4e47-477d-a751-e1c9c20e0e7c-tls-cert\") pod \"authorino-687bd45585-2jpsk\" (UID: \"444ed13f-4e47-477d-a751-e1c9c20e0e7c\") " pod="kuadrant-system/authorino-687bd45585-2jpsk"
Sep 08 11:39:36.284354 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.284326    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvvz\" (UniqueName: \"kubernetes.io/projected/444ed13f-4e47-477d-a751-e1c9c20e0e7c-kube-api-access-tjvvz\") pod \"authorino-687bd45585-2jpsk\" (UID: \"444ed13f-4e47-477d-a751-e1c9c20e0e7c\") " pod="kuadrant-system/authorino-687bd45585-2jpsk"
Sep 08 11:39:36.375351 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.375306    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvqkr\" (UniqueName: \"kubernetes.io/projected/10a17813-dcee-42e7-a38c-ab434db3fbdc-kube-api-access-mvqkr\") pod \"10a17813-dcee-42e7-a38c-ab434db3fbdc\" (UID: \"10a17813-dcee-42e7-a38c-ab434db3fbdc\") "
Sep 08 11:39:36.375559 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.375367    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjvvz\" (UniqueName: \"kubernetes.io/projected/444ed13f-4e47-477d-a751-e1c9c20e0e7c-kube-api-access-tjvvz\") pod \"444ed13f-4e47-477d-a751-e1c9c20e0e7c\" (UID: \"444ed13f-4e47-477d-a751-e1c9c20e0e7c\") "
Sep 08 11:39:36.375559 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.375452    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/444ed13f-4e47-477d-a751-e1c9c20e0e7c-tls-cert\") pod \"444ed13f-4e47-477d-a751-e1c9c20e0e7c\" (UID: \"444ed13f-4e47-477d-a751-e1c9c20e0e7c\") "
Sep 08 11:39:36.375688 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.375626    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b767f0b0-a09a-4224-a59e-5a1e5282cdea-tls-cert\") pod \"authorino-6bfd87fdb4-6t9w5\" (UID: \"b767f0b0-a09a-4224-a59e-5a1e5282cdea\") " pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5"
Sep 08 11:39:36.375770 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.375739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcsll\" (UniqueName: \"kubernetes.io/projected/b767f0b0-a09a-4224-a59e-5a1e5282cdea-kube-api-access-wcsll\") pod \"authorino-6bfd87fdb4-6t9w5\" (UID: \"b767f0b0-a09a-4224-a59e-5a1e5282cdea\") " pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5"
Sep 08 11:39:36.377539 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.377505    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a17813-dcee-42e7-a38c-ab434db3fbdc-kube-api-access-mvqkr" (OuterVolumeSpecName: "kube-api-access-mvqkr") pod "10a17813-dcee-42e7-a38c-ab434db3fbdc" (UID: "10a17813-dcee-42e7-a38c-ab434db3fbdc"). InnerVolumeSpecName "kube-api-access-mvqkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:39:36.377539 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.377513    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/444ed13f-4e47-477d-a751-e1c9c20e0e7c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "444ed13f-4e47-477d-a751-e1c9c20e0e7c" (UID: "444ed13f-4e47-477d-a751-e1c9c20e0e7c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:39:36.377719 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.377557    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444ed13f-4e47-477d-a751-e1c9c20e0e7c-kube-api-access-tjvvz" (OuterVolumeSpecName: "kube-api-access-tjvvz") pod "444ed13f-4e47-477d-a751-e1c9c20e0e7c" (UID: "444ed13f-4e47-477d-a751-e1c9c20e0e7c"). InnerVolumeSpecName "kube-api-access-tjvvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:39:36.476369 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.476329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b767f0b0-a09a-4224-a59e-5a1e5282cdea-tls-cert\") pod \"authorino-6bfd87fdb4-6t9w5\" (UID: \"b767f0b0-a09a-4224-a59e-5a1e5282cdea\") " pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5"
Sep 08 11:39:36.476571 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.476396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcsll\" (UniqueName: \"kubernetes.io/projected/b767f0b0-a09a-4224-a59e-5a1e5282cdea-kube-api-access-wcsll\") pod \"authorino-6bfd87fdb4-6t9w5\" (UID: \"b767f0b0-a09a-4224-a59e-5a1e5282cdea\") " pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5"
Sep 08 11:39:36.476571 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.476450    2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/444ed13f-4e47-477d-a751-e1c9c20e0e7c-tls-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:39:36.476571 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.476460    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvqkr\" (UniqueName: \"kubernetes.io/projected/10a17813-dcee-42e7-a38c-ab434db3fbdc-kube-api-access-mvqkr\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:39:36.476571 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.476487    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjvvz\" (UniqueName: \"kubernetes.io/projected/444ed13f-4e47-477d-a751-e1c9c20e0e7c-kube-api-access-tjvvz\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:39:36.478859 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.478835    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b767f0b0-a09a-4224-a59e-5a1e5282cdea-tls-cert\") pod \"authorino-6bfd87fdb4-6t9w5\" (UID: \"b767f0b0-a09a-4224-a59e-5a1e5282cdea\") " pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5"
Sep 08 11:39:36.485634 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.485601    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcsll\" (UniqueName: \"kubernetes.io/projected/b767f0b0-a09a-4224-a59e-5a1e5282cdea-kube-api-access-wcsll\") pod \"authorino-6bfd87fdb4-6t9w5\" (UID: \"b767f0b0-a09a-4224-a59e-5a1e5282cdea\") " pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5"
Sep 08 11:39:36.509583 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.509556    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5"
Sep 08 11:39:36.778615 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:36.778585    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bfd87fdb4-6t9w5"]
Sep 08 11:39:36.781008 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:39:36.780979    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb767f0b0_a09a_4224_a59e_5a1e5282cdea.slice/crio-35c61887825eed5b05a11732766a384aa615f2c7bbb9ba914d21435de4459cd5 WatchSource:0}: Error finding container 35c61887825eed5b05a11732766a384aa615f2c7bbb9ba914d21435de4459cd5: Status 404 returned error can't find the container with id 35c61887825eed5b05a11732766a384aa615f2c7bbb9ba914d21435de4459cd5
Sep 08 11:39:37.212535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:37.212501    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5" event={"ID":"b767f0b0-a09a-4224-a59e-5a1e5282cdea","Type":"ContainerStarted","Data":"35c61887825eed5b05a11732766a384aa615f2c7bbb9ba914d21435de4459cd5"}
Sep 08 11:39:37.212696 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:37.212579    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-swl9t"
Sep 08 11:39:37.212696 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:37.212580    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-2jpsk"
Sep 08 11:39:37.251202 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:37.251167    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-swl9t"]
Sep 08 11:39:37.255976 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:37.255945    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-swl9t"]
Sep 08 11:39:37.286383 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:37.286337    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-2jpsk"]
Sep 08 11:39:37.288875 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:37.288841    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-2jpsk"]
Sep 08 11:39:37.965484 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:37.965442    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10a17813-dcee-42e7-a38c-ab434db3fbdc" path="/var/lib/kubelet/pods/10a17813-dcee-42e7-a38c-ab434db3fbdc/volumes"
Sep 08 11:39:37.965929 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:37.965906    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="444ed13f-4e47-477d-a751-e1c9c20e0e7c" path="/var/lib/kubelet/pods/444ed13f-4e47-477d-a751-e1c9c20e0e7c/volumes"
Sep 08 11:39:38.219207 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:38.219118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5" event={"ID":"b767f0b0-a09a-4224-a59e-5a1e5282cdea","Type":"ContainerStarted","Data":"fbb7444021e10893a6f926d357366309d36f3a650bd1db36dc0ead4b7720c1a6"}
Sep 08 11:39:38.238145 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:38.238084    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5" podStartSLOduration=1.7277910300000001 podStartE2EDuration="2.238068339s" totalImagesPullingTime="510.277309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:39:38.237179289 +0000 UTC m=+584.803939832" watchObservedRunningTime="2026-09-08 11:39:38.238068339 +0000 UTC m=+584.804828880"
Sep 08 11:39:38.268615 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:38.268584    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-p9d4z"]
Sep 08 11:39:38.268792 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:38.268771    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-p9d4z" podUID="caf7bdb7-ef63-41fb-8c93-87ca879864e7" containerName="authorino" containerID="cri-o://0a52d74d0d06cd06eea8d956d4aaf49a06de17a2ada7e48256288a568904a12e" gracePeriod=30
Sep 08 11:39:38.512135 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:38.512108    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-p9d4z"
Sep 08 11:39:38.695871 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:38.695831    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvnt5\" (UniqueName: \"kubernetes.io/projected/caf7bdb7-ef63-41fb-8c93-87ca879864e7-kube-api-access-hvnt5\") pod \"caf7bdb7-ef63-41fb-8c93-87ca879864e7\" (UID: \"caf7bdb7-ef63-41fb-8c93-87ca879864e7\") "
Sep 08 11:39:38.697965 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:38.697926    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf7bdb7-ef63-41fb-8c93-87ca879864e7-kube-api-access-hvnt5" (OuterVolumeSpecName: "kube-api-access-hvnt5") pod "caf7bdb7-ef63-41fb-8c93-87ca879864e7" (UID: "caf7bdb7-ef63-41fb-8c93-87ca879864e7"). InnerVolumeSpecName "kube-api-access-hvnt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:39:38.796641 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:38.796552    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvnt5\" (UniqueName: \"kubernetes.io/projected/caf7bdb7-ef63-41fb-8c93-87ca879864e7-kube-api-access-hvnt5\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:39:39.223855 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:39.223820    2561 generic.go:356] "Generic (PLEG): container finished" podID="caf7bdb7-ef63-41fb-8c93-87ca879864e7" containerID="0a52d74d0d06cd06eea8d956d4aaf49a06de17a2ada7e48256288a568904a12e" exitCode=0
Sep 08 11:39:39.224316 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:39.223860    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-p9d4z"
Sep 08 11:39:39.224316 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:39.223914    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-p9d4z" event={"ID":"caf7bdb7-ef63-41fb-8c93-87ca879864e7","Type":"ContainerDied","Data":"0a52d74d0d06cd06eea8d956d4aaf49a06de17a2ada7e48256288a568904a12e"}
Sep 08 11:39:39.224316 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:39.223960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-p9d4z" event={"ID":"caf7bdb7-ef63-41fb-8c93-87ca879864e7","Type":"ContainerDied","Data":"9dc3b7d61b626e781625401345a64e1f91e03c437eb343d7ba0b208af58e72ca"}
Sep 08 11:39:39.224316 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:39.223986    2561 scope.go:117] "RemoveContainer" containerID="0a52d74d0d06cd06eea8d956d4aaf49a06de17a2ada7e48256288a568904a12e"
Sep 08 11:39:39.232068 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:39.232047    2561 scope.go:117] "RemoveContainer" containerID="0a52d74d0d06cd06eea8d956d4aaf49a06de17a2ada7e48256288a568904a12e"
Sep 08 11:39:39.232332 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:39:39.232311    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a52d74d0d06cd06eea8d956d4aaf49a06de17a2ada7e48256288a568904a12e\": container with ID starting with 0a52d74d0d06cd06eea8d956d4aaf49a06de17a2ada7e48256288a568904a12e not found: ID does not exist" containerID="0a52d74d0d06cd06eea8d956d4aaf49a06de17a2ada7e48256288a568904a12e"
Sep 08 11:39:39.232409 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:39.232345    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a52d74d0d06cd06eea8d956d4aaf49a06de17a2ada7e48256288a568904a12e"} err="failed to get container status \"0a52d74d0d06cd06eea8d956d4aaf49a06de17a2ada7e48256288a568904a12e\": rpc error: code = NotFound desc = could not find container \"0a52d74d0d06cd06eea8d956d4aaf49a06de17a2ada7e48256288a568904a12e\": container with ID starting with 0a52d74d0d06cd06eea8d956d4aaf49a06de17a2ada7e48256288a568904a12e not found: ID does not exist"
Sep 08 11:39:39.245262 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:39.245236    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-p9d4z"]
Sep 08 11:39:39.252402 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:39.252379    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-p9d4z"]
Sep 08 11:39:39.959815 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:39.959782    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caf7bdb7-ef63-41fb-8c93-87ca879864e7" path="/var/lib/kubelet/pods/caf7bdb7-ef63-41fb-8c93-87ca879864e7/volumes"
Sep 08 11:39:43.035541 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.035507    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:39:43.065677 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.065637    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cb48d57d-prb4p"]
Sep 08 11:39:43.066056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.066042    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf7bdb7-ef63-41fb-8c93-87ca879864e7" containerName="authorino"
Sep 08 11:39:43.066111 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.066059    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf7bdb7-ef63-41fb-8c93-87ca879864e7" containerName="authorino"
Sep 08 11:39:43.066178 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.066131    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf7bdb7-ef63-41fb-8c93-87ca879864e7" containerName="authorino"
Sep 08 11:39:43.070748 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.070727    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:43.073370 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.073340    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 11:39:43.073523 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.073347    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-z8t97\""
Sep 08 11:39:43.073523 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.073506    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 11:39:43.074511 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.074487    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 11:39:43.085506 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.084337    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb48d57d-prb4p"]
Sep 08 11:39:43.127955 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.127922    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-metrics-certs\") pod \"maas-controller-6cb48d57d-prb4p\" (UID: \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\") " pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:43.128128 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.127965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j97rh\" (UniqueName: \"kubernetes.io/projected/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-kube-api-access-j97rh\") pod \"maas-controller-6cb48d57d-prb4p\" (UID: \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\") " pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:43.128128 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.127993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-webhook-cert\") pod \"maas-controller-6cb48d57d-prb4p\" (UID: \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\") " pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:43.140871 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.140833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:39:43.229058 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.229017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-metrics-certs\") pod \"maas-controller-6cb48d57d-prb4p\" (UID: \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\") " pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:43.229252 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.229089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j97rh\" (UniqueName: \"kubernetes.io/projected/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-kube-api-access-j97rh\") pod \"maas-controller-6cb48d57d-prb4p\" (UID: \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\") " pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:43.229252 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.229134    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-webhook-cert\") pod \"maas-controller-6cb48d57d-prb4p\" (UID: \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\") " pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:43.229252 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:39:43.229217    2561 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 08 11:39:43.229378 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:39:43.229321    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-webhook-cert podName:1f3f470a-00bd-49ee-8b0b-0b7ceb90a552 nodeName:}" failed. No retries permitted until 2026-09-08 11:39:43.729299811 +0000 UTC m=+590.296060350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-webhook-cert") pod "maas-controller-6cb48d57d-prb4p" (UID: "1f3f470a-00bd-49ee-8b0b-0b7ceb90a552") : secret "maas-controller-webhook-cert" not found
Sep 08 11:39:43.231761 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.231730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-metrics-certs\") pod \"maas-controller-6cb48d57d-prb4p\" (UID: \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\") " pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:43.232791 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.232765    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:39:43.233303 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.233282    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-764984dc4c-5jmzf"]
Sep 08 11:39:43.236480 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.236452    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:39:43.241045 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.241023    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j97rh\" (UniqueName: \"kubernetes.io/projected/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-kube-api-access-j97rh\") pod \"maas-controller-6cb48d57d-prb4p\" (UID: \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\") " pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:43.247106 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.246863    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-764984dc4c-5jmzf"]
Sep 08 11:39:43.330013 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.329908    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/293bf1e0-f877-4b8d-b306-ab7d01b0974f-webhook-cert\") pod \"maas-controller-764984dc4c-5jmzf\" (UID: \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\") " pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:39:43.330013 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.329960    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnn4s\" (UniqueName: \"kubernetes.io/projected/293bf1e0-f877-4b8d-b306-ab7d01b0974f-kube-api-access-hnn4s\") pod \"maas-controller-764984dc4c-5jmzf\" (UID: \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\") " pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:39:43.330193 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.330089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/293bf1e0-f877-4b8d-b306-ab7d01b0974f-metrics-certs\") pod \"maas-controller-764984dc4c-5jmzf\" (UID: \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\") " pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:39:43.332292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.332266    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:39:43.430897 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.430858    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/293bf1e0-f877-4b8d-b306-ab7d01b0974f-metrics-certs\") pod \"maas-controller-764984dc4c-5jmzf\" (UID: \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\") " pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:39:43.431098 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.430902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/293bf1e0-f877-4b8d-b306-ab7d01b0974f-webhook-cert\") pod \"maas-controller-764984dc4c-5jmzf\" (UID: \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\") " pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:39:43.431098 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.430930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnn4s\" (UniqueName: \"kubernetes.io/projected/293bf1e0-f877-4b8d-b306-ab7d01b0974f-kube-api-access-hnn4s\") pod \"maas-controller-764984dc4c-5jmzf\" (UID: \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\") " pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:39:43.433339 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.433313    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/293bf1e0-f877-4b8d-b306-ab7d01b0974f-webhook-cert\") pod \"maas-controller-764984dc4c-5jmzf\" (UID: \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\") " pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:39:43.433460 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.433343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/293bf1e0-f877-4b8d-b306-ab7d01b0974f-metrics-certs\") pod \"maas-controller-764984dc4c-5jmzf\" (UID: \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\") " pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:39:43.440220 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.440196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnn4s\" (UniqueName: \"kubernetes.io/projected/293bf1e0-f877-4b8d-b306-ab7d01b0974f-kube-api-access-hnn4s\") pod \"maas-controller-764984dc4c-5jmzf\" (UID: \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\") " pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:39:43.451085 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.451060    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:39:43.556812 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.556777    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:39:43.734038 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.734000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-webhook-cert\") pod \"maas-controller-6cb48d57d-prb4p\" (UID: \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\") " pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:43.736362 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.736342    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-webhook-cert\") pod \"maas-controller-6cb48d57d-prb4p\" (UID: \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\") " pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:43.829784 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.829758    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-764984dc4c-5jmzf"]
Sep 08 11:39:43.832242 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:39:43.832217    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293bf1e0_f877_4b8d_b306_ab7d01b0974f.slice/crio-10c79c5963ae4817b4ff4a9020900d7e4e72fc3d023e611e5d40c58710d82cd4 WatchSource:0}: Error finding container 10c79c5963ae4817b4ff4a9020900d7e4e72fc3d023e611e5d40c58710d82cd4: Status 404 returned error can't find the container with id 10c79c5963ae4817b4ff4a9020900d7e4e72fc3d023e611e5d40c58710d82cd4
Sep 08 11:39:43.982322 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:43.982288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:44.247156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:44.247085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764984dc4c-5jmzf" event={"ID":"293bf1e0-f877-4b8d-b306-ab7d01b0974f","Type":"ContainerStarted","Data":"10c79c5963ae4817b4ff4a9020900d7e4e72fc3d023e611e5d40c58710d82cd4"}
Sep 08 11:39:44.265817 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:44.265788    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb48d57d-prb4p"]
Sep 08 11:39:44.268892 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:39:44.268867    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3f470a_00bd_49ee_8b0b_0b7ceb90a552.slice/crio-50865651bdd9a38c07f62565c89fc5f583b56c705599560a9cbaad8092d0cd36 WatchSource:0}: Error finding container 50865651bdd9a38c07f62565c89fc5f583b56c705599560a9cbaad8092d0cd36: Status 404 returned error can't find the container with id 50865651bdd9a38c07f62565c89fc5f583b56c705599560a9cbaad8092d0cd36
Sep 08 11:39:45.255946 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:45.255897    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb48d57d-prb4p" event={"ID":"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552","Type":"ContainerStarted","Data":"50865651bdd9a38c07f62565c89fc5f583b56c705599560a9cbaad8092d0cd36"}
Sep 08 11:39:46.824195 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:46.824149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:39:46.924073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:46.923991    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:39:47.266864 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:47.266822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb48d57d-prb4p" event={"ID":"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552","Type":"ContainerStarted","Data":"34648c5822629fdf29f5fa02e91da22ad2dbab6e5784fb579fc28b7a96fae577"}
Sep 08 11:39:47.267051 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:47.266918    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:47.269069 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:47.269040    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764984dc4c-5jmzf" event={"ID":"293bf1e0-f877-4b8d-b306-ab7d01b0974f","Type":"ContainerStarted","Data":"48223244ec9d38f67fccb4c562f8c94404cb85516991b46d066cb4f70df12866"}
Sep 08 11:39:47.269246 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:47.269222    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:39:47.291962 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:47.291911    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cb48d57d-prb4p" podStartSLOduration=1.92421748 podStartE2EDuration="4.291896074s" totalImagesPullingTime="2.367678594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:39:47.291004329 +0000 UTC m=+593.857764871" watchObservedRunningTime="2026-09-08 11:39:47.291896074 +0000 UTC m=+593.858656615"
Sep 08 11:39:47.321849 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:47.321792    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-764984dc4c-5jmzf" podStartSLOduration=1.527206496 podStartE2EDuration="4.321775054s" totalImagesPullingTime="2.794568558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:39:47.320631574 +0000 UTC m=+593.887392116" watchObservedRunningTime="2026-09-08 11:39:47.321775054 +0000 UTC m=+593.888535613"
Sep 08 11:39:58.276406 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.276369    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:39:58.276829 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.276531    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:58.347527 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.347498    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb48d57d-prb4p"]
Sep 08 11:39:58.347714 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.347676    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6cb48d57d-prb4p" podUID="1f3f470a-00bd-49ee-8b0b-0b7ceb90a552" containerName="manager" containerID="cri-o://34648c5822629fdf29f5fa02e91da22ad2dbab6e5784fb579fc28b7a96fae577" gracePeriod=30
Sep 08 11:39:58.586877 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.586849    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:58.646875 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.646836    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-metrics-certs\") pod \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\" (UID: \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\") "
Sep 08 11:39:58.647036 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.646923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-webhook-cert\") pod \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\" (UID: \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\") "
Sep 08 11:39:58.647036 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.646950    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j97rh\" (UniqueName: \"kubernetes.io/projected/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-kube-api-access-j97rh\") pod \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\" (UID: \"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552\") "
Sep 08 11:39:58.649060 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.649027    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f3f470a-00bd-49ee-8b0b-0b7ceb90a552" (UID: "1f3f470a-00bd-49ee-8b0b-0b7ceb90a552"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:39:58.649060 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.649053    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1f3f470a-00bd-49ee-8b0b-0b7ceb90a552" (UID: "1f3f470a-00bd-49ee-8b0b-0b7ceb90a552"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:39:58.649214 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.649077    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-kube-api-access-j97rh" (OuterVolumeSpecName: "kube-api-access-j97rh") pod "1f3f470a-00bd-49ee-8b0b-0b7ceb90a552" (UID: "1f3f470a-00bd-49ee-8b0b-0b7ceb90a552"). InnerVolumeSpecName "kube-api-access-j97rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:39:58.747765 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.747721    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-metrics-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:39:58.747765 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.747762    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-webhook-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:39:58.747969 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:58.747775    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j97rh\" (UniqueName: \"kubernetes.io/projected/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552-kube-api-access-j97rh\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:39:59.319634 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:59.319599    2561 generic.go:356] "Generic (PLEG): container finished" podID="1f3f470a-00bd-49ee-8b0b-0b7ceb90a552" containerID="34648c5822629fdf29f5fa02e91da22ad2dbab6e5784fb579fc28b7a96fae577" exitCode=0
Sep 08 11:39:59.320040 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:59.319645    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb48d57d-prb4p"
Sep 08 11:39:59.320040 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:59.319676    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb48d57d-prb4p" event={"ID":"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552","Type":"ContainerDied","Data":"34648c5822629fdf29f5fa02e91da22ad2dbab6e5784fb579fc28b7a96fae577"}
Sep 08 11:39:59.320040 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:59.319714    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb48d57d-prb4p" event={"ID":"1f3f470a-00bd-49ee-8b0b-0b7ceb90a552","Type":"ContainerDied","Data":"50865651bdd9a38c07f62565c89fc5f583b56c705599560a9cbaad8092d0cd36"}
Sep 08 11:39:59.320040 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:59.319735    2561 scope.go:117] "RemoveContainer" containerID="34648c5822629fdf29f5fa02e91da22ad2dbab6e5784fb579fc28b7a96fae577"
Sep 08 11:39:59.333187 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:59.333164    2561 scope.go:117] "RemoveContainer" containerID="34648c5822629fdf29f5fa02e91da22ad2dbab6e5784fb579fc28b7a96fae577"
Sep 08 11:39:59.333497 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:39:59.333458    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34648c5822629fdf29f5fa02e91da22ad2dbab6e5784fb579fc28b7a96fae577\": container with ID starting with 34648c5822629fdf29f5fa02e91da22ad2dbab6e5784fb579fc28b7a96fae577 not found: ID does not exist" containerID="34648c5822629fdf29f5fa02e91da22ad2dbab6e5784fb579fc28b7a96fae577"
Sep 08 11:39:59.333594 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:59.333503    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34648c5822629fdf29f5fa02e91da22ad2dbab6e5784fb579fc28b7a96fae577"} err="failed to get container status \"34648c5822629fdf29f5fa02e91da22ad2dbab6e5784fb579fc28b7a96fae577\": rpc error: code = NotFound desc = could not find container \"34648c5822629fdf29f5fa02e91da22ad2dbab6e5784fb579fc28b7a96fae577\": container with ID starting with 34648c5822629fdf29f5fa02e91da22ad2dbab6e5784fb579fc28b7a96fae577 not found: ID does not exist"
Sep 08 11:39:59.347748 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:59.347713    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb48d57d-prb4p"]
Sep 08 11:39:59.352813 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:59.352782    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cb48d57d-prb4p"]
Sep 08 11:39:59.960279 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:39:59.960245    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f3f470a-00bd-49ee-8b0b-0b7ceb90a552" path="/var/lib/kubelet/pods/1f3f470a-00bd-49ee-8b0b-0b7ceb90a552/volumes"
Sep 08 11:40:18.460009 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:18.459920    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:40:18.562032 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:18.561999    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:40:19.861555 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:19.861519    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:40:19.959853 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:19.959814    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:40:20.310396 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.310342    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:40:20.526089 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.526050    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:40:20.625349 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.625264    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:40:20.636401 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.636373    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c"]
Sep 08 11:40:20.636749 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.636706    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3f470a-00bd-49ee-8b0b-0b7ceb90a552" containerName="manager"
Sep 08 11:40:20.636815 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.636752    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3f470a-00bd-49ee-8b0b-0b7ceb90a552" containerName="manager"
Sep 08 11:40:20.636815 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.636805    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3f470a-00bd-49ee-8b0b-0b7ceb90a552" containerName="manager"
Sep 08 11:40:20.639677 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.639656    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c"
Sep 08 11:40:20.642379 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.642359    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 08 11:40:20.653159 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.653134    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c"]
Sep 08 11:40:20.719288 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.719250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxh5j\" (UniqueName: \"kubernetes.io/projected/1bf3b339-f292-4de7-94f4-6e80cf89001e-kube-api-access-dxh5j\") pod \"payload-pre-processing-59fbc47bf8-wmp6c\" (UID: \"1bf3b339-f292-4de7-94f4-6e80cf89001e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c"
Sep 08 11:40:20.719492 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.719340    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf3b339-f292-4de7-94f4-6e80cf89001e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wmp6c\" (UID: \"1bf3b339-f292-4de7-94f4-6e80cf89001e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c"
Sep 08 11:40:20.819846 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.819799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxh5j\" (UniqueName: \"kubernetes.io/projected/1bf3b339-f292-4de7-94f4-6e80cf89001e-kube-api-access-dxh5j\") pod \"payload-pre-processing-59fbc47bf8-wmp6c\" (UID: \"1bf3b339-f292-4de7-94f4-6e80cf89001e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c"
Sep 08 11:40:20.820043 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.819863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf3b339-f292-4de7-94f4-6e80cf89001e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wmp6c\" (UID: \"1bf3b339-f292-4de7-94f4-6e80cf89001e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c"
Sep 08 11:40:20.820589 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.820567    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf3b339-f292-4de7-94f4-6e80cf89001e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wmp6c\" (UID: \"1bf3b339-f292-4de7-94f4-6e80cf89001e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c"
Sep 08 11:40:20.829037 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.829001    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxh5j\" (UniqueName: \"kubernetes.io/projected/1bf3b339-f292-4de7-94f4-6e80cf89001e-kube-api-access-dxh5j\") pod \"payload-pre-processing-59fbc47bf8-wmp6c\" (UID: \"1bf3b339-f292-4de7-94f4-6e80cf89001e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c"
Sep 08 11:40:20.950224 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:20.950131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c"
Sep 08 11:40:21.223513 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:21.223458    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c"]
Sep 08 11:40:21.410125 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:21.410085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c" event={"ID":"1bf3b339-f292-4de7-94f4-6e80cf89001e","Type":"ContainerStarted","Data":"b743ea92a6c9a82773bc3e3c73cdea680bbcbca2a84085b380a5283b7e80d32f"}
Sep 08 11:40:24.425710 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:24.425663    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c" event={"ID":"1bf3b339-f292-4de7-94f4-6e80cf89001e","Type":"ContainerStarted","Data":"594b321a7f77d3ee34fed9c8c89f9f4685e08a483b6402f5fa482b4c2bb15b17"}
Sep 08 11:40:24.426093 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:24.425758    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c"
Sep 08 11:40:24.445266 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:24.445201    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c" podStartSLOduration=1.922519452 podStartE2EDuration="4.445186831s" totalImagesPullingTime="2.522667379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:40:24.444195096 +0000 UTC m=+631.010955637" watchObservedRunningTime="2026-09-08 11:40:24.445186831 +0000 UTC m=+631.011947375"
Sep 08 11:40:33.624227 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:33.624190    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:40:33.713993 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:33.713959    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:40:35.431665 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:35.431629    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wmp6c"
Sep 08 11:40:39.525545 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.525510    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"]
Sep 08 11:40:39.528878 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.528853    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.531451 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.531424    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 11:40:39.531616 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.531590    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 08 11:40:39.531616 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.531606    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 11:40:39.532651 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.532636    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pkvgk\""
Sep 08 11:40:39.539752 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.539731    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"]
Sep 08 11:40:39.546119 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.546094    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:40:39.678895 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.678855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d184a11a-9283-4db7-a600-7c3216d6f046-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.679077 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.678909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d184a11a-9283-4db7-a600-7c3216d6f046-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.679077 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.678986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d184a11a-9283-4db7-a600-7c3216d6f046-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.679077 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.679016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqjqk\" (UniqueName: \"kubernetes.io/projected/d184a11a-9283-4db7-a600-7c3216d6f046-kube-api-access-bqjqk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.679077 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.679042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d184a11a-9283-4db7-a600-7c3216d6f046-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.679249 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.679117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d184a11a-9283-4db7-a600-7c3216d6f046-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.780198 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.780094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d184a11a-9283-4db7-a600-7c3216d6f046-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.780198 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.780154    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d184a11a-9283-4db7-a600-7c3216d6f046-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.780198 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.780187    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d184a11a-9283-4db7-a600-7c3216d6f046-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.780502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.780216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d184a11a-9283-4db7-a600-7c3216d6f046-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.780502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.780233    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqjqk\" (UniqueName: \"kubernetes.io/projected/d184a11a-9283-4db7-a600-7c3216d6f046-kube-api-access-bqjqk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.780502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.780254    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d184a11a-9283-4db7-a600-7c3216d6f046-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.780677 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.780606    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d184a11a-9283-4db7-a600-7c3216d6f046-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.780756 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.780731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d184a11a-9283-4db7-a600-7c3216d6f046-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.780813 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.780738    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d184a11a-9283-4db7-a600-7c3216d6f046-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.782582 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.782558    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d184a11a-9283-4db7-a600-7c3216d6f046-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.782812 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.782796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d184a11a-9283-4db7-a600-7c3216d6f046-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.789450 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.789429    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqjqk\" (UniqueName: \"kubernetes.io/projected/d184a11a-9283-4db7-a600-7c3216d6f046-kube-api-access-bqjqk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn\" (UID: \"d184a11a-9283-4db7-a600-7c3216d6f046\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:39.839536 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:39.839460    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:40.119931 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:40.119855    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"]
Sep 08 11:40:40.123839 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:40:40.123809    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd184a11a_9283_4db7_a600_7c3216d6f046.slice/crio-f1bdbafc18c5337caa1bc4be3719ac6b9a5a1def2a75f45e52f6ee31c78f4fe8 WatchSource:0}: Error finding container f1bdbafc18c5337caa1bc4be3719ac6b9a5a1def2a75f45e52f6ee31c78f4fe8: Status 404 returned error can't find the container with id f1bdbafc18c5337caa1bc4be3719ac6b9a5a1def2a75f45e52f6ee31c78f4fe8
Sep 08 11:40:40.499617 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:40.499570    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn" event={"ID":"d184a11a-9283-4db7-a600-7c3216d6f046","Type":"ContainerStarted","Data":"f1bdbafc18c5337caa1bc4be3719ac6b9a5a1def2a75f45e52f6ee31c78f4fe8"}
Sep 08 11:40:41.505357 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:41.505321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn" event={"ID":"d184a11a-9283-4db7-a600-7c3216d6f046","Type":"ContainerStarted","Data":"79783f05632dc8db55245e584b3003335cf92969ef5c4c4cabaf73e13b454a12"}
Sep 08 11:40:41.531098 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:41.531046    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn" podStartSLOduration=1.287964146 podStartE2EDuration="2.531028616s" totalImagesPullingTime="1.24306447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:40:41.530530249 +0000 UTC m=+648.097290791" watchObservedRunningTime="2026-09-08 11:40:41.531028616 +0000 UTC m=+648.097789170"
Sep 08 11:40:42.427037 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.427000    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"]
Sep 08 11:40:42.430289 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.430268    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.432833 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.432810    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 08 11:40:42.441390 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.441369    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"]
Sep 08 11:40:42.459490 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.459452    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:40:42.604590 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.604557    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79fe915f-6658-4171-b290-0b1741204169-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.604590 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.604592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79fe915f-6658-4171-b290-0b1741204169-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.605037 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.604714    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79fe915f-6658-4171-b290-0b1741204169-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.605037 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.604763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79fe915f-6658-4171-b290-0b1741204169-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.605037 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.604794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcwq6\" (UniqueName: \"kubernetes.io/projected/79fe915f-6658-4171-b290-0b1741204169-kube-api-access-dcwq6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.605037 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.604876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79fe915f-6658-4171-b290-0b1741204169-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.706295 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.706265    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79fe915f-6658-4171-b290-0b1741204169-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.706511 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.706302    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79fe915f-6658-4171-b290-0b1741204169-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.706511 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.706322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79fe915f-6658-4171-b290-0b1741204169-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.706511 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.706340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79fe915f-6658-4171-b290-0b1741204169-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.706511 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.706359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcwq6\" (UniqueName: \"kubernetes.io/projected/79fe915f-6658-4171-b290-0b1741204169-kube-api-access-dcwq6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.706733 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.706553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79fe915f-6658-4171-b290-0b1741204169-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.706860 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.706839    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79fe915f-6658-4171-b290-0b1741204169-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.706974 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.706868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79fe915f-6658-4171-b290-0b1741204169-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.706974 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.706898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79fe915f-6658-4171-b290-0b1741204169-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.708505 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.708487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79fe915f-6658-4171-b290-0b1741204169-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.708835 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.708815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79fe915f-6658-4171-b290-0b1741204169-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.716650 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.716629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcwq6\" (UniqueName: \"kubernetes.io/projected/79fe915f-6658-4171-b290-0b1741204169-kube-api-access-dcwq6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vlm68\" (UID: \"79fe915f-6658-4171-b290-0b1741204169\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:42.740585 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:42.740553    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:43.016405 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:43.016323    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"]
Sep 08 11:40:43.019497 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:40:43.019446    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fe915f_6658_4171_b290_0b1741204169.slice/crio-982c0e4c1568f0dba319bbb28b79e0b74a70b10f8b18b3ba26b9ae8082e63c8e WatchSource:0}: Error finding container 982c0e4c1568f0dba319bbb28b79e0b74a70b10f8b18b3ba26b9ae8082e63c8e: Status 404 returned error can't find the container with id 982c0e4c1568f0dba319bbb28b79e0b74a70b10f8b18b3ba26b9ae8082e63c8e
Sep 08 11:40:43.020986 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:43.020966    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 11:40:43.514874 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:43.514843    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68" event={"ID":"79fe915f-6658-4171-b290-0b1741204169","Type":"ContainerStarted","Data":"862f98e45416c99666fd395071ec5eec6a5c80d32d8780a1a45fcf5dac24c793"}
Sep 08 11:40:43.515041 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:43.514892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68" event={"ID":"79fe915f-6658-4171-b290-0b1741204169","Type":"ContainerStarted","Data":"982c0e4c1568f0dba319bbb28b79e0b74a70b10f8b18b3ba26b9ae8082e63c8e"}
Sep 08 11:40:43.536932 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:43.536869    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68" podStartSLOduration=1.346519976 podStartE2EDuration="1.536853531s" totalImagesPullingTime="190.333555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:40:43.535911864 +0000 UTC m=+650.102672406" watchObservedRunningTime="2026-09-08 11:40:43.536853531 +0000 UTC m=+650.103614072"
Sep 08 11:40:49.840454 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:49.840419    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:49.840843 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:49.840499    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:49.853071 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:49.853045    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:50.554792 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:50.554762    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn"
Sep 08 11:40:52.740964 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:52.740909    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:52.740964 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:52.740982    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:52.753721 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:52.753689    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:53.567728 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:53.567701    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vlm68"
Sep 08 11:40:54.152457 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:40:54.152420    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:41:05.771085 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:05.771052    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:41:12.063431 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:12.063393    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:41:37.532313 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:37.532268    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:41:39.620116 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.620073    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"]
Sep 08 11:41:39.623285 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.623264    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.625933 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.625905    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 08 11:41:39.632037 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.632013    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"]
Sep 08 11:41:39.755700 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.755661    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.755891 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.755711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.755891 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.755808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh4td\" (UniqueName: \"kubernetes.io/projected/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-kube-api-access-kh4td\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.755891 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.755869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.756062 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.755926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.756062 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.755978    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.856602 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.856569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.856766 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.856617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.856766 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.856649    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.856766 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.856684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.856766 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.856708    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.856766 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.856737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh4td\" (UniqueName: \"kubernetes.io/projected/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-kube-api-access-kh4td\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.857042 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.857016    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.857116 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.857053    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.857171 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.857130    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.858932 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.858911    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.859219 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.859200    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.866336 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.866311    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh4td\" (UniqueName: \"kubernetes.io/projected/fbd4cd91-16a7-4c1a-9444-833b6f13ca72-kube-api-access-kh4td\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-52k84\" (UID: \"fbd4cd91-16a7-4c1a-9444-833b6f13ca72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:39.932865 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:39.932774    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:40.215641 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:40.215574    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"]
Sep 08 11:41:40.217789 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:41:40.217758    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbd4cd91_16a7_4c1a_9444_833b6f13ca72.slice/crio-804b0f7bb10d5a7a04e362d332e025825a196467c8ae59200c36538a04c8cf9a WatchSource:0}: Error finding container 804b0f7bb10d5a7a04e362d332e025825a196467c8ae59200c36538a04c8cf9a: Status 404 returned error can't find the container with id 804b0f7bb10d5a7a04e362d332e025825a196467c8ae59200c36538a04c8cf9a
Sep 08 11:41:40.748961 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:40.748928    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84" event={"ID":"fbd4cd91-16a7-4c1a-9444-833b6f13ca72","Type":"ContainerStarted","Data":"2b0a544cd4cd6ec63f7a73e885c2b62154949624e27c38690f3e1c7af7208ab7"}
Sep 08 11:41:40.749358 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:40.748973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84" event={"ID":"fbd4cd91-16a7-4c1a-9444-833b6f13ca72","Type":"ContainerStarted","Data":"804b0f7bb10d5a7a04e362d332e025825a196467c8ae59200c36538a04c8cf9a"}
Sep 08 11:41:40.767669 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:40.767621    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84" podStartSLOduration=1.587662699 podStartE2EDuration="1.76760402s" totalImagesPullingTime="179.941321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:41:40.767385432 +0000 UTC m=+707.334145972" watchObservedRunningTime="2026-09-08 11:41:40.76760402 +0000 UTC m=+707.334364562"
Sep 08 11:41:41.142519 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:41.142386    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:41:49.065739 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:49.065646    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:41:49.166179 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:49.166149    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:41:49.933814 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:49.933774    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:49.933814 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:49.933827    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:49.946597 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:49.946565    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:41:50.801154 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:41:50.801124    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-52k84"
Sep 08 11:42:11.494508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:11.494453    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:11.562699 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:11.562667    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:31.288879 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:31.288848    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-549d4746c7-6jlvm"]
Sep 08 11:42:31.292024 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:31.292005    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-549d4746c7-6jlvm"
Sep 08 11:42:31.300083 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:31.300053    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-549d4746c7-6jlvm"]
Sep 08 11:42:31.383044 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:31.382998    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/940ded72-0df4-44fc-8ebd-7bc974ea730e-tls-cert\") pod \"authorino-549d4746c7-6jlvm\" (UID: \"940ded72-0df4-44fc-8ebd-7bc974ea730e\") " pod="kuadrant-system/authorino-549d4746c7-6jlvm"
Sep 08 11:42:31.383044 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:31.383043    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c46w7\" (UniqueName: \"kubernetes.io/projected/940ded72-0df4-44fc-8ebd-7bc974ea730e-kube-api-access-c46w7\") pod \"authorino-549d4746c7-6jlvm\" (UID: \"940ded72-0df4-44fc-8ebd-7bc974ea730e\") " pod="kuadrant-system/authorino-549d4746c7-6jlvm"
Sep 08 11:42:31.483498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:31.483446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/940ded72-0df4-44fc-8ebd-7bc974ea730e-tls-cert\") pod \"authorino-549d4746c7-6jlvm\" (UID: \"940ded72-0df4-44fc-8ebd-7bc974ea730e\") " pod="kuadrant-system/authorino-549d4746c7-6jlvm"
Sep 08 11:42:31.483498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:31.483497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c46w7\" (UniqueName: \"kubernetes.io/projected/940ded72-0df4-44fc-8ebd-7bc974ea730e-kube-api-access-c46w7\") pod \"authorino-549d4746c7-6jlvm\" (UID: \"940ded72-0df4-44fc-8ebd-7bc974ea730e\") " pod="kuadrant-system/authorino-549d4746c7-6jlvm"
Sep 08 11:42:31.485963 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:31.485935    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/940ded72-0df4-44fc-8ebd-7bc974ea730e-tls-cert\") pod \"authorino-549d4746c7-6jlvm\" (UID: \"940ded72-0df4-44fc-8ebd-7bc974ea730e\") " pod="kuadrant-system/authorino-549d4746c7-6jlvm"
Sep 08 11:42:31.492601 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:31.492579    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c46w7\" (UniqueName: \"kubernetes.io/projected/940ded72-0df4-44fc-8ebd-7bc974ea730e-kube-api-access-c46w7\") pod \"authorino-549d4746c7-6jlvm\" (UID: \"940ded72-0df4-44fc-8ebd-7bc974ea730e\") " pod="kuadrant-system/authorino-549d4746c7-6jlvm"
Sep 08 11:42:31.601880 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:31.601783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-549d4746c7-6jlvm"
Sep 08 11:42:31.871592 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:31.871554    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-549d4746c7-6jlvm"]
Sep 08 11:42:31.874395 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:42:31.874366    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod940ded72_0df4_44fc_8ebd_7bc974ea730e.slice/crio-3b9c2c9934b9177558284f8145458f71248dfe37efc216ba38066e2307402e84 WatchSource:0}: Error finding container 3b9c2c9934b9177558284f8145458f71248dfe37efc216ba38066e2307402e84: Status 404 returned error can't find the container with id 3b9c2c9934b9177558284f8145458f71248dfe37efc216ba38066e2307402e84
Sep 08 11:42:31.967607 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:31.967574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-549d4746c7-6jlvm" event={"ID":"940ded72-0df4-44fc-8ebd-7bc974ea730e","Type":"ContainerStarted","Data":"3b9c2c9934b9177558284f8145458f71248dfe37efc216ba38066e2307402e84"}
Sep 08 11:42:32.976044 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:32.975947    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-549d4746c7-6jlvm" event={"ID":"940ded72-0df4-44fc-8ebd-7bc974ea730e","Type":"ContainerStarted","Data":"ff22c65a0597f50a12e2472e10c4d88789600eb0139707f242ff5eed9f37a48b"}
Sep 08 11:42:32.996038 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:32.995976    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-549d4746c7-6jlvm" podStartSLOduration=1.152692499 podStartE2EDuration="1.995954756s" totalImagesPullingTime="843.262257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:42:32.995678 +0000 UTC m=+759.562438541" watchObservedRunningTime="2026-09-08 11:42:32.995954756 +0000 UTC m=+759.562715298"
Sep 08 11:42:33.026528 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.026493    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bfd87fdb4-6t9w5"]
Sep 08 11:42:33.026715 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.026680    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5" podUID="b767f0b0-a09a-4224-a59e-5a1e5282cdea" containerName="authorino" containerID="cri-o://fbb7444021e10893a6f926d357366309d36f3a650bd1db36dc0ead4b7720c1a6" gracePeriod=30
Sep 08 11:42:33.270037 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.270012    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5"
Sep 08 11:42:33.399884 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.399845    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b767f0b0-a09a-4224-a59e-5a1e5282cdea-tls-cert\") pod \"b767f0b0-a09a-4224-a59e-5a1e5282cdea\" (UID: \"b767f0b0-a09a-4224-a59e-5a1e5282cdea\") "
Sep 08 11:42:33.400048 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.399923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcsll\" (UniqueName: \"kubernetes.io/projected/b767f0b0-a09a-4224-a59e-5a1e5282cdea-kube-api-access-wcsll\") pod \"b767f0b0-a09a-4224-a59e-5a1e5282cdea\" (UID: \"b767f0b0-a09a-4224-a59e-5a1e5282cdea\") "
Sep 08 11:42:33.401914 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.401879    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b767f0b0-a09a-4224-a59e-5a1e5282cdea-kube-api-access-wcsll" (OuterVolumeSpecName: "kube-api-access-wcsll") pod "b767f0b0-a09a-4224-a59e-5a1e5282cdea" (UID: "b767f0b0-a09a-4224-a59e-5a1e5282cdea"). InnerVolumeSpecName "kube-api-access-wcsll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:42:33.410052 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.410013    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b767f0b0-a09a-4224-a59e-5a1e5282cdea-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b767f0b0-a09a-4224-a59e-5a1e5282cdea" (UID: "b767f0b0-a09a-4224-a59e-5a1e5282cdea"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:42:33.501188 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.501142    2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b767f0b0-a09a-4224-a59e-5a1e5282cdea-tls-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:42:33.501188 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.501187    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcsll\" (UniqueName: \"kubernetes.io/projected/b767f0b0-a09a-4224-a59e-5a1e5282cdea-kube-api-access-wcsll\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:42:33.530724 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.530641    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:33.630428 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.630381    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:33.982545 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.982516    2561 generic.go:356] "Generic (PLEG): container finished" podID="b767f0b0-a09a-4224-a59e-5a1e5282cdea" containerID="fbb7444021e10893a6f926d357366309d36f3a650bd1db36dc0ead4b7720c1a6" exitCode=0
Sep 08 11:42:33.982946 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.982562    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5"
Sep 08 11:42:33.982946 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.982597    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5" event={"ID":"b767f0b0-a09a-4224-a59e-5a1e5282cdea","Type":"ContainerDied","Data":"fbb7444021e10893a6f926d357366309d36f3a650bd1db36dc0ead4b7720c1a6"}
Sep 08 11:42:33.982946 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.982639    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bfd87fdb4-6t9w5" event={"ID":"b767f0b0-a09a-4224-a59e-5a1e5282cdea","Type":"ContainerDied","Data":"35c61887825eed5b05a11732766a384aa615f2c7bbb9ba914d21435de4459cd5"}
Sep 08 11:42:33.982946 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.982658    2561 scope.go:117] "RemoveContainer" containerID="fbb7444021e10893a6f926d357366309d36f3a650bd1db36dc0ead4b7720c1a6"
Sep 08 11:42:33.991018 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.990998    2561 scope.go:117] "RemoveContainer" containerID="fbb7444021e10893a6f926d357366309d36f3a650bd1db36dc0ead4b7720c1a6"
Sep 08 11:42:33.991349 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:42:33.991321    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb7444021e10893a6f926d357366309d36f3a650bd1db36dc0ead4b7720c1a6\": container with ID starting with fbb7444021e10893a6f926d357366309d36f3a650bd1db36dc0ead4b7720c1a6 not found: ID does not exist" containerID="fbb7444021e10893a6f926d357366309d36f3a650bd1db36dc0ead4b7720c1a6"
Sep 08 11:42:33.991427 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:33.991352    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb7444021e10893a6f926d357366309d36f3a650bd1db36dc0ead4b7720c1a6"} err="failed to get container status \"fbb7444021e10893a6f926d357366309d36f3a650bd1db36dc0ead4b7720c1a6\": rpc error: code = NotFound desc = could not find container \"fbb7444021e10893a6f926d357366309d36f3a650bd1db36dc0ead4b7720c1a6\": container with ID starting with fbb7444021e10893a6f926d357366309d36f3a650bd1db36dc0ead4b7720c1a6 not found: ID does not exist"
Sep 08 11:42:34.008660 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:34.008630    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bfd87fdb4-6t9w5"]
Sep 08 11:42:34.012876 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:34.012847    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6bfd87fdb4-6t9w5"]
Sep 08 11:42:34.653769 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:34.653732    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:34.755164 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:34.755125    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:35.960255 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:35.960217    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b767f0b0-a09a-4224-a59e-5a1e5282cdea" path="/var/lib/kubelet/pods/b767f0b0-a09a-4224-a59e-5a1e5282cdea/volumes"
Sep 08 11:42:51.110404 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:51.110368    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:51.224199 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:51.224167    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:51.471755 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:51.471723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:51.714358 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:51.714329    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:52.629580 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:52.629536    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:52.738323 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:52.738291    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:53.071160 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:53.071128    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7449-140146733610160"]
Sep 08 11:42:53.071431 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:53.071419    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b767f0b0-a09a-4224-a59e-5a1e5282cdea" containerName="authorino"
Sep 08 11:42:53.071499 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:53.071432    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b767f0b0-a09a-4224-a59e-5a1e5282cdea" containerName="authorino"
Sep 08 11:42:53.071538 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:53.071503    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b767f0b0-a09a-4224-a59e-5a1e5282cdea" containerName="authorino"
Sep 08 11:42:53.074201 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:53.074184    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7449-140146733610160"
Sep 08 11:42:53.076975 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:53.076955    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zbjsb\""
Sep 08 11:42:53.086005 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:53.085980    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7449-140146733610160"]
Sep 08 11:42:53.178921 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:53.178883    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jqm7\" (UniqueName: \"kubernetes.io/projected/990f490f-0787-47fa-b4cd-0f30d1a1e744-kube-api-access-2jqm7\") pod \"test-curl-7449-140146733610160\" (UID: \"990f490f-0787-47fa-b4cd-0f30d1a1e744\") " pod="opendatahub/test-curl-7449-140146733610160"
Sep 08 11:42:53.280251 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:53.280214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jqm7\" (UniqueName: \"kubernetes.io/projected/990f490f-0787-47fa-b4cd-0f30d1a1e744-kube-api-access-2jqm7\") pod \"test-curl-7449-140146733610160\" (UID: \"990f490f-0787-47fa-b4cd-0f30d1a1e744\") " pod="opendatahub/test-curl-7449-140146733610160"
Sep 08 11:42:53.289438 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:53.289417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jqm7\" (UniqueName: \"kubernetes.io/projected/990f490f-0787-47fa-b4cd-0f30d1a1e744-kube-api-access-2jqm7\") pod \"test-curl-7449-140146733610160\" (UID: \"990f490f-0787-47fa-b4cd-0f30d1a1e744\") " pod="opendatahub/test-curl-7449-140146733610160"
Sep 08 11:42:53.382964 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:53.382872    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7449-140146733610160"
Sep 08 11:42:53.651883 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:53.651858    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7449-140146733610160"]
Sep 08 11:42:53.653876 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:42:53.653850    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod990f490f_0787_47fa_b4cd_0f30d1a1e744.slice/crio-0a5caf93cc47750008409dca96f52c1d76b2c6058caeffb428b5531770712bed WatchSource:0}: Error finding container 0a5caf93cc47750008409dca96f52c1d76b2c6058caeffb428b5531770712bed: Status 404 returned error can't find the container with id 0a5caf93cc47750008409dca96f52c1d76b2c6058caeffb428b5531770712bed
Sep 08 11:42:54.067025 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:54.066976    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7449-140146733610160" event={"ID":"990f490f-0787-47fa-b4cd-0f30d1a1e744","Type":"ContainerStarted","Data":"0a5caf93cc47750008409dca96f52c1d76b2c6058caeffb428b5531770712bed"}
Sep 08 11:42:55.073699 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:55.073663    2561 generic.go:356] "Generic (PLEG): container finished" podID="990f490f-0787-47fa-b4cd-0f30d1a1e744" containerID="40d41afc2cafe286b0c1b4beeaf35e75b19b339ee4b35f596a169ea6d9d6eb72" exitCode=0
Sep 08 11:42:55.074060 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:55.073730    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7449-140146733610160" event={"ID":"990f490f-0787-47fa-b4cd-0f30d1a1e744","Type":"ContainerDied","Data":"40d41afc2cafe286b0c1b4beeaf35e75b19b339ee4b35f596a169ea6d9d6eb72"}
Sep 08 11:42:56.120276 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.120240    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"]
Sep 08 11:42:56.123612 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.123592    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"
Sep 08 11:42:56.126416 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.126394    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-95669b\""
Sep 08 11:42:56.127017 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.127002    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-95669b-dockercfg-w7s49\""
Sep 08 11:42:56.135882 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.135861    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"]
Sep 08 11:42:56.149070 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.149043    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"]
Sep 08 11:42:56.152525 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.152503    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"
Sep 08 11:42:56.155495 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.155449    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:56.163892 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.163870    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"]
Sep 08 11:42:56.208868 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.208842    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7449-140146733610160"
Sep 08 11:42:56.229970 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.229939    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7449-140146733610160_990f490f-0787-47fa-b4cd-0f30d1a1e744/test-curl-7449-140146733610160/0.log"
Sep 08 11:42:56.230985 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.230961    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:56.253515 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.253481    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7449-140146733610160"]
Sep 08 11:42:56.258753 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.258727    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7449-140146733610160"]
Sep 08 11:42:56.304609 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.304579    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jqm7\" (UniqueName: \"kubernetes.io/projected/990f490f-0787-47fa-b4cd-0f30d1a1e744-kube-api-access-2jqm7\") pod \"990f490f-0787-47fa-b4cd-0f30d1a1e744\" (UID: \"990f490f-0787-47fa-b4cd-0f30d1a1e744\") "
Sep 08 11:42:56.304789 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.304721    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r8vn\" (UniqueName: \"kubernetes.io/projected/a631a187-cc9d-4e74-ab8c-028b6013ff02-kube-api-access-6r8vn\") pod \"payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p\" (UID: \"a631a187-cc9d-4e74-ab8c-028b6013ff02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"
Sep 08 11:42:56.304789 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.304745    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmrbr\" (UniqueName: \"kubernetes.io/projected/d93e1465-3497-40b4-a67f-7232c883c545-kube-api-access-dmrbr\") pod \"payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k\" (UID: \"d93e1465-3497-40b4-a67f-7232c883c545\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"
Sep 08 11:42:56.304903 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.304788    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a631a187-cc9d-4e74-ab8c-028b6013ff02-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p\" (UID: \"a631a187-cc9d-4e74-ab8c-028b6013ff02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"
Sep 08 11:42:56.304903 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.304861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93e1465-3497-40b4-a67f-7232c883c545-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k\" (UID: \"d93e1465-3497-40b4-a67f-7232c883c545\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"
Sep 08 11:42:56.306830 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.306803    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990f490f-0787-47fa-b4cd-0f30d1a1e744-kube-api-access-2jqm7" (OuterVolumeSpecName: "kube-api-access-2jqm7") pod "990f490f-0787-47fa-b4cd-0f30d1a1e744" (UID: "990f490f-0787-47fa-b4cd-0f30d1a1e744"). InnerVolumeSpecName "kube-api-access-2jqm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:42:56.405718 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.405635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a631a187-cc9d-4e74-ab8c-028b6013ff02-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p\" (UID: \"a631a187-cc9d-4e74-ab8c-028b6013ff02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"
Sep 08 11:42:56.405718 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.405685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93e1465-3497-40b4-a67f-7232c883c545-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k\" (UID: \"d93e1465-3497-40b4-a67f-7232c883c545\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"
Sep 08 11:42:56.405932 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.405746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r8vn\" (UniqueName: \"kubernetes.io/projected/a631a187-cc9d-4e74-ab8c-028b6013ff02-kube-api-access-6r8vn\") pod \"payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p\" (UID: \"a631a187-cc9d-4e74-ab8c-028b6013ff02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"
Sep 08 11:42:56.405932 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.405772    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrbr\" (UniqueName: \"kubernetes.io/projected/d93e1465-3497-40b4-a67f-7232c883c545-kube-api-access-dmrbr\") pod \"payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k\" (UID: \"d93e1465-3497-40b4-a67f-7232c883c545\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"
Sep 08 11:42:56.405932 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.405815    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jqm7\" (UniqueName: \"kubernetes.io/projected/990f490f-0787-47fa-b4cd-0f30d1a1e744-kube-api-access-2jqm7\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:42:56.406364 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.406344    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93e1465-3497-40b4-a67f-7232c883c545-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k\" (UID: \"d93e1465-3497-40b4-a67f-7232c883c545\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"
Sep 08 11:42:56.406422 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.406402    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a631a187-cc9d-4e74-ab8c-028b6013ff02-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p\" (UID: \"a631a187-cc9d-4e74-ab8c-028b6013ff02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"
Sep 08 11:42:56.415317 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.415293    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrbr\" (UniqueName: \"kubernetes.io/projected/d93e1465-3497-40b4-a67f-7232c883c545-kube-api-access-dmrbr\") pod \"payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k\" (UID: \"d93e1465-3497-40b4-a67f-7232c883c545\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"
Sep 08 11:42:56.415426 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.415354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r8vn\" (UniqueName: \"kubernetes.io/projected/a631a187-cc9d-4e74-ab8c-028b6013ff02-kube-api-access-6r8vn\") pod \"payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p\" (UID: \"a631a187-cc9d-4e74-ab8c-028b6013ff02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"
Sep 08 11:42:56.435352 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.435327    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"
Sep 08 11:42:56.465874 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.465843    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"
Sep 08 11:42:56.711639 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.708935    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"]
Sep 08 11:42:56.746817 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:56.746791    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"]
Sep 08 11:42:56.750218 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:42:56.750179    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda631a187_cc9d_4e74_ab8c_028b6013ff02.slice/crio-3ca979517c216ccbb0f00d39a96be5d9982759168e0d97d8baf7e461098e0cf2 WatchSource:0}: Error finding container 3ca979517c216ccbb0f00d39a96be5d9982759168e0d97d8baf7e461098e0cf2: Status 404 returned error can't find the container with id 3ca979517c216ccbb0f00d39a96be5d9982759168e0d97d8baf7e461098e0cf2
Sep 08 11:42:57.085971 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:57.085869    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k" event={"ID":"d93e1465-3497-40b4-a67f-7232c883c545","Type":"ContainerStarted","Data":"98a42b455d0b56d43637817f02c0525ce0152eea1d241bb6ea090268b3e4f54d"}
Sep 08 11:42:57.085971 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:57.085932    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k" event={"ID":"d93e1465-3497-40b4-a67f-7232c883c545","Type":"ContainerStarted","Data":"e8c00c94df64d5bb406986465d64283195b5951a2dd5a1f1c23c8623cb6f66da"}
Sep 08 11:42:57.086191 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:57.085980    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"
Sep 08 11:42:57.087493 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:57.087448    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7449-140146733610160"
Sep 08 11:42:57.087493 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:57.087492    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a5caf93cc47750008409dca96f52c1d76b2c6058caeffb428b5531770712bed"
Sep 08 11:42:57.089153 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:57.089129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p" event={"ID":"a631a187-cc9d-4e74-ab8c-028b6013ff02","Type":"ContainerStarted","Data":"3416999acfa4c811c631103362e4fd2a1bcb3438e43d6c00516ebb2c1f660c7d"}
Sep 08 11:42:57.089280 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:57.089174    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p" event={"ID":"a631a187-cc9d-4e74-ab8c-028b6013ff02","Type":"ContainerStarted","Data":"3ca979517c216ccbb0f00d39a96be5d9982759168e0d97d8baf7e461098e0cf2"}
Sep 08 11:42:57.089410 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:57.089281    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"
Sep 08 11:42:57.104785 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:57.104740    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k" podStartSLOduration=1.104725187 podStartE2EDuration="1.104725187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:42:57.104229529 +0000 UTC m=+783.670990071" watchObservedRunningTime="2026-09-08 11:42:57.104725187 +0000 UTC m=+783.671485727"
Sep 08 11:42:57.124198 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:57.124150    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p" podStartSLOduration=1.124135367 podStartE2EDuration="1.124135367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:42:57.123376077 +0000 UTC m=+783.690136619" watchObservedRunningTime="2026-09-08 11:42:57.124135367 +0000 UTC m=+783.690895907"
Sep 08 11:42:57.125776 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:42:57.125750    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7449-140146733610160\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="990f490f-0787-47fa-b4cd-0f30d1a1e744" pod="opendatahub/test-curl-7449-140146733610160"
Sep 08 11:42:57.960906 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:57.960868    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="990f490f-0787-47fa-b4cd-0f30d1a1e744" path="/var/lib/kubelet/pods/990f490f-0787-47fa-b4cd-0f30d1a1e744/volumes"
Sep 08 11:42:59.815674 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:59.815633    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:42:59.919348 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:42:59.919311    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:00.056578 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:00.056534    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:00.367915 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:00.367864    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:00.740373 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:00.740322    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:01.548947 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:01.548912    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:01.862574 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:01.862420    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:02.004181 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:02.004132    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:02.779328 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:02.779285    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:02.880644 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:02.880612    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:03.942885 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:03.942852    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:04.023691 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.023656    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:04.133691 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.133658    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:04.556223 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.556188    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:04.652022 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.651987    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"]
Sep 08 11:43:04.652309 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.652297    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="990f490f-0787-47fa-b4cd-0f30d1a1e744" containerName="test-curl-7449-140146733610160"
Sep 08 11:43:04.652363 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.652310    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="990f490f-0787-47fa-b4cd-0f30d1a1e744" containerName="test-curl-7449-140146733610160"
Sep 08 11:43:04.652398 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.652386    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="990f490f-0787-47fa-b4cd-0f30d1a1e744" containerName="test-curl-7449-140146733610160"
Sep 08 11:43:04.655204 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.655170    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:04.656460 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.656446    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"
Sep 08 11:43:04.658967 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.658947    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7d3a6048\""
Sep 08 11:43:04.666985 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.666960    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"]
Sep 08 11:43:04.679715 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.679690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6348aba7-6439-4400-80ef-29ac41a2bac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc\" (UID: \"6348aba7-6439-4400-80ef-29ac41a2bac6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"
Sep 08 11:43:04.679842 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.679746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxrpn\" (UniqueName: \"kubernetes.io/projected/6348aba7-6439-4400-80ef-29ac41a2bac6-kube-api-access-mxrpn\") pod \"payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc\" (UID: \"6348aba7-6439-4400-80ef-29ac41a2bac6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"
Sep 08 11:43:04.693170 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.693139    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"]
Sep 08 11:43:04.696369 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.696355    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"
Sep 08 11:43:04.698783 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.698766    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7d3a6048-dockercfg-bshc6\""
Sep 08 11:43:04.706208 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.706184    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"]
Sep 08 11:43:04.780132 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.780097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6348aba7-6439-4400-80ef-29ac41a2bac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc\" (UID: \"6348aba7-6439-4400-80ef-29ac41a2bac6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"
Sep 08 11:43:04.780132 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.780137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2vf\" (UniqueName: \"kubernetes.io/projected/041e8061-0ede-4a51-94a7-10be919ba3be-kube-api-access-kp2vf\") pod \"payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z\" (UID: \"041e8061-0ede-4a51-94a7-10be919ba3be\") " pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"
Sep 08 11:43:04.780399 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.780185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxrpn\" (UniqueName: \"kubernetes.io/projected/6348aba7-6439-4400-80ef-29ac41a2bac6-kube-api-access-mxrpn\") pod \"payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc\" (UID: \"6348aba7-6439-4400-80ef-29ac41a2bac6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"
Sep 08 11:43:04.780399 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.780222    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041e8061-0ede-4a51-94a7-10be919ba3be-plugins-config-volume\") pod \"payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z\" (UID: \"041e8061-0ede-4a51-94a7-10be919ba3be\") " pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"
Sep 08 11:43:04.780929 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.780905    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6348aba7-6439-4400-80ef-29ac41a2bac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc\" (UID: \"6348aba7-6439-4400-80ef-29ac41a2bac6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"
Sep 08 11:43:04.789210 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.789177    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxrpn\" (UniqueName: \"kubernetes.io/projected/6348aba7-6439-4400-80ef-29ac41a2bac6-kube-api-access-mxrpn\") pod \"payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc\" (UID: \"6348aba7-6439-4400-80ef-29ac41a2bac6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"
Sep 08 11:43:04.881248 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.881166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041e8061-0ede-4a51-94a7-10be919ba3be-plugins-config-volume\") pod \"payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z\" (UID: \"041e8061-0ede-4a51-94a7-10be919ba3be\") " pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"
Sep 08 11:43:04.881248 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.881221    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2vf\" (UniqueName: \"kubernetes.io/projected/041e8061-0ede-4a51-94a7-10be919ba3be-kube-api-access-kp2vf\") pod \"payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z\" (UID: \"041e8061-0ede-4a51-94a7-10be919ba3be\") " pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"
Sep 08 11:43:04.881893 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.881873    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041e8061-0ede-4a51-94a7-10be919ba3be-plugins-config-volume\") pod \"payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z\" (UID: \"041e8061-0ede-4a51-94a7-10be919ba3be\") " pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"
Sep 08 11:43:04.889916 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.889892    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2vf\" (UniqueName: \"kubernetes.io/projected/041e8061-0ede-4a51-94a7-10be919ba3be-kube-api-access-kp2vf\") pod \"payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z\" (UID: \"041e8061-0ede-4a51-94a7-10be919ba3be\") " pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"
Sep 08 11:43:04.967274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:04.967237    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"
Sep 08 11:43:05.007038 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:05.006805    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"
Sep 08 11:43:05.245802 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:05.245765    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"]
Sep 08 11:43:05.249026 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:43:05.249002    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6348aba7_6439_4400_80ef_29ac41a2bac6.slice/crio-1ffeb873e20cdf3fdb114c64229a6802a72c8da05ae2b91f017db528594b1034 WatchSource:0}: Error finding container 1ffeb873e20cdf3fdb114c64229a6802a72c8da05ae2b91f017db528594b1034: Status 404 returned error can't find the container with id 1ffeb873e20cdf3fdb114c64229a6802a72c8da05ae2b91f017db528594b1034
Sep 08 11:43:05.287569 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:05.287540    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"]
Sep 08 11:43:05.290242 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:43:05.290216    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041e8061_0ede_4a51_94a7_10be919ba3be.slice/crio-3b12c0dc89d10a2cbc39d48ca3bdb4ca5687a08cc313924056c08c0e6641b2aa WatchSource:0}: Error finding container 3b12c0dc89d10a2cbc39d48ca3bdb4ca5687a08cc313924056c08c0e6641b2aa: Status 404 returned error can't find the container with id 3b12c0dc89d10a2cbc39d48ca3bdb4ca5687a08cc313924056c08c0e6641b2aa
Sep 08 11:43:06.131427 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:06.131385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z" event={"ID":"041e8061-0ede-4a51-94a7-10be919ba3be","Type":"ContainerStarted","Data":"64c1417e6b88d2716db7c0863c75f01e946b00305229e010c099956ad2d3ea22"}
Sep 08 11:43:06.131909 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:06.131451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z" event={"ID":"041e8061-0ede-4a51-94a7-10be919ba3be","Type":"ContainerStarted","Data":"3b12c0dc89d10a2cbc39d48ca3bdb4ca5687a08cc313924056c08c0e6641b2aa"}
Sep 08 11:43:06.131909 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:06.131494    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"
Sep 08 11:43:06.133118 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:06.133095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc" event={"ID":"6348aba7-6439-4400-80ef-29ac41a2bac6","Type":"ContainerStarted","Data":"455cdb078f984bd09c0fc5a80c13ca29d028ee799d1d22f3c818527595df5c36"}
Sep 08 11:43:06.133207 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:06.133139    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc" event={"ID":"6348aba7-6439-4400-80ef-29ac41a2bac6","Type":"ContainerStarted","Data":"1ffeb873e20cdf3fdb114c64229a6802a72c8da05ae2b91f017db528594b1034"}
Sep 08 11:43:06.133207 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:06.133166    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"
Sep 08 11:43:06.155969 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:06.155908    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z" podStartSLOduration=2.155882382 podStartE2EDuration="2.155882382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:43:06.154600759 +0000 UTC m=+792.721361311" watchObservedRunningTime="2026-09-08 11:43:06.155882382 +0000 UTC m=+792.722642925"
Sep 08 11:43:08.095651 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:08.095616    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"
Sep 08 11:43:08.096013 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:08.095919    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"
Sep 08 11:43:08.115124 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:08.115072    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc" podStartSLOduration=4.1150567 podStartE2EDuration="4.1150567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:43:06.178312088 +0000 UTC m=+792.745072630" watchObservedRunningTime="2026-09-08 11:43:08.1150567 +0000 UTC m=+794.681817241"
Sep 08 11:43:09.434091 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:09.434057    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:09.530538 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:09.530504    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:09.765270 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:09.765231    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:09.938699 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:09.938659    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:10.048201 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:10.048120    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:10.153786 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:10.153752    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:10.244098 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:10.244067    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:10.343697 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:10.343613    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:10.673528 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:10.673430    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:10.769367 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:10.769333    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:11.010458 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.010423    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:11.371692 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.371602    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:11.477443 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.477408    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:11.555641 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.555602    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"]
Sep 08 11:43:11.559216 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.559193    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"
Sep 08 11:43:11.561723 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.561703    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a1721d\""
Sep 08 11:43:11.561812 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.561745    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-a1721d-dockercfg-wjpvf\""
Sep 08 11:43:11.572183 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.572158    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"]
Sep 08 11:43:11.638738 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.638649    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427ac17c-5f57-4a92-b88e-02428119fb80-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr\" (UID: \"427ac17c-5f57-4a92-b88e-02428119fb80\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"
Sep 08 11:43:11.638738 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.638705    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rghjc\" (UniqueName: \"kubernetes.io/projected/427ac17c-5f57-4a92-b88e-02428119fb80-kube-api-access-rghjc\") pod \"payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr\" (UID: \"427ac17c-5f57-4a92-b88e-02428119fb80\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"
Sep 08 11:43:11.665524 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.665488    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"]
Sep 08 11:43:11.668786 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.668764    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"
Sep 08 11:43:11.682427 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.682402    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"]
Sep 08 11:43:11.739817 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.739779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427ac17c-5f57-4a92-b88e-02428119fb80-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr\" (UID: \"427ac17c-5f57-4a92-b88e-02428119fb80\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"
Sep 08 11:43:11.739962 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.739843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec32d74-55da-43c6-804d-ef8976022e60-plugins-config-volume\") pod \"payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89\" (UID: \"fec32d74-55da-43c6-804d-ef8976022e60\") " pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"
Sep 08 11:43:11.739962 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.739903    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rghjc\" (UniqueName: \"kubernetes.io/projected/427ac17c-5f57-4a92-b88e-02428119fb80-kube-api-access-rghjc\") pod \"payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr\" (UID: \"427ac17c-5f57-4a92-b88e-02428119fb80\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"
Sep 08 11:43:11.740071 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.739999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6js4h\" (UniqueName: \"kubernetes.io/projected/fec32d74-55da-43c6-804d-ef8976022e60-kube-api-access-6js4h\") pod \"payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89\" (UID: \"fec32d74-55da-43c6-804d-ef8976022e60\") " pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"
Sep 08 11:43:11.740509 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.740493    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427ac17c-5f57-4a92-b88e-02428119fb80-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr\" (UID: \"427ac17c-5f57-4a92-b88e-02428119fb80\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"
Sep 08 11:43:11.750076 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.750045    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rghjc\" (UniqueName: \"kubernetes.io/projected/427ac17c-5f57-4a92-b88e-02428119fb80-kube-api-access-rghjc\") pod \"payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr\" (UID: \"427ac17c-5f57-4a92-b88e-02428119fb80\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"
Sep 08 11:43:11.840627 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.840588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6js4h\" (UniqueName: \"kubernetes.io/projected/fec32d74-55da-43c6-804d-ef8976022e60-kube-api-access-6js4h\") pod \"payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89\" (UID: \"fec32d74-55da-43c6-804d-ef8976022e60\") " pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"
Sep 08 11:43:11.840833 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.840655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec32d74-55da-43c6-804d-ef8976022e60-plugins-config-volume\") pod \"payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89\" (UID: \"fec32d74-55da-43c6-804d-ef8976022e60\") " pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"
Sep 08 11:43:11.841399 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.841377    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec32d74-55da-43c6-804d-ef8976022e60-plugins-config-volume\") pod \"payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89\" (UID: \"fec32d74-55da-43c6-804d-ef8976022e60\") " pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"
Sep 08 11:43:11.852879 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.852850    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6js4h\" (UniqueName: \"kubernetes.io/projected/fec32d74-55da-43c6-804d-ef8976022e60-kube-api-access-6js4h\") pod \"payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89\" (UID: \"fec32d74-55da-43c6-804d-ef8976022e60\") " pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"
Sep 08 11:43:11.869599 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.869568    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"
Sep 08 11:43:11.979329 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:11.979294    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"
Sep 08 11:43:12.151110 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:12.151051    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"]
Sep 08 11:43:12.163510 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:12.163483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr" event={"ID":"427ac17c-5f57-4a92-b88e-02428119fb80","Type":"ContainerStarted","Data":"80678c4e72f00c810bbc1f4005c6c31b650324ff677fe130e9f870c4587f7eea"}
Sep 08 11:43:12.270498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:12.270456    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"]
Sep 08 11:43:12.273606 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:43:12.273577    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec32d74_55da_43c6_804d_ef8976022e60.slice/crio-fb93b712d5a5aad008ff0b66fd0a0855f2b599d92b4f0f55616fc4014ce21f94 WatchSource:0}: Error finding container fb93b712d5a5aad008ff0b66fd0a0855f2b599d92b4f0f55616fc4014ce21f94: Status 404 returned error can't find the container with id fb93b712d5a5aad008ff0b66fd0a0855f2b599d92b4f0f55616fc4014ce21f94
Sep 08 11:43:12.895478 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:12.895439    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:13.170938 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:13.170830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr" event={"ID":"427ac17c-5f57-4a92-b88e-02428119fb80","Type":"ContainerStarted","Data":"4593188b2a415cc93c31d478e66383b7b5bba07dfadab1f4800e1516b4b9fd92"}
Sep 08 11:43:13.170938 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:13.170903    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"
Sep 08 11:43:13.172580 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:13.172549    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89" event={"ID":"fec32d74-55da-43c6-804d-ef8976022e60","Type":"ContainerStarted","Data":"eb267fd1de6b240fa3ee2ef6ef8bab9969dcc5c4653f82f854fd9aa7ea0564f6"}
Sep 08 11:43:13.172706 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:13.172605    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89" event={"ID":"fec32d74-55da-43c6-804d-ef8976022e60","Type":"ContainerStarted","Data":"fb93b712d5a5aad008ff0b66fd0a0855f2b599d92b4f0f55616fc4014ce21f94"}
Sep 08 11:43:13.172706 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:13.172677    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"
Sep 08 11:43:13.198758 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:13.198703    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr" podStartSLOduration=2.198685293 podStartE2EDuration="2.198685293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:43:13.19682243 +0000 UTC m=+799.763582973" watchObservedRunningTime="2026-09-08 11:43:13.198685293 +0000 UTC m=+799.765445833"
Sep 08 11:43:13.214482 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:13.214408    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89" podStartSLOduration=2.214393161 podStartE2EDuration="2.214393161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:43:13.213157606 +0000 UTC m=+799.779918169" watchObservedRunningTime="2026-09-08 11:43:13.214393161 +0000 UTC m=+799.781153701"
Sep 08 11:43:14.547851 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:14.547811    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:16.439900 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:16.439815    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:17.140039 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:17.140007    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"
Sep 08 11:43:17.140198 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:17.140093    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"
Sep 08 11:43:17.888323 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:17.888286    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:17.989766 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:17.989736    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:18.089421 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:18.089389    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:18.272003 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:18.271950    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:19.311445 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:19.311411    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:19.645139 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:19.645056    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:20.013691 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:20.013648    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:20.161897 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:20.161868    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:23.595927 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:23.595888    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:23.695993 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:23.695950    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:24.178414 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:24.178383    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"
Sep 08 11:43:24.178619 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:24.178592    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"
Sep 08 11:43:24.951130 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:24.951091    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:25.025127 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.025092    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:25.496993 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.496962    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:25.594001 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.593963    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:25.615419 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.615392    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"]
Sep 08 11:43:25.618972 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.618942    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"
Sep 08 11:43:25.621726 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.621707    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-425b2c\""
Sep 08 11:43:25.621853 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.621770    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-425b2c-dockercfg-pxg82\""
Sep 08 11:43:25.631187 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.631165    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"]
Sep 08 11:43:25.666253 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.666217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec97bdfa-8926-4c5c-9924-d06d84b68cbe-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c\" (UID: \"ec97bdfa-8926-4c5c-9924-d06d84b68cbe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"
Sep 08 11:43:25.666253 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.666250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwsw7\" (UniqueName: \"kubernetes.io/projected/ec97bdfa-8926-4c5c-9924-d06d84b68cbe-kube-api-access-zwsw7\") pod \"payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c\" (UID: \"ec97bdfa-8926-4c5c-9924-d06d84b68cbe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"
Sep 08 11:43:25.678937 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.678908    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:25.767436 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.767361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec97bdfa-8926-4c5c-9924-d06d84b68cbe-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c\" (UID: \"ec97bdfa-8926-4c5c-9924-d06d84b68cbe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"
Sep 08 11:43:25.767436 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.767396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwsw7\" (UniqueName: \"kubernetes.io/projected/ec97bdfa-8926-4c5c-9924-d06d84b68cbe-kube-api-access-zwsw7\") pod \"payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c\" (UID: \"ec97bdfa-8926-4c5c-9924-d06d84b68cbe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"
Sep 08 11:43:25.768125 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.768103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec97bdfa-8926-4c5c-9924-d06d84b68cbe-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c\" (UID: \"ec97bdfa-8926-4c5c-9924-d06d84b68cbe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"
Sep 08 11:43:25.779714 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.779691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwsw7\" (UniqueName: \"kubernetes.io/projected/ec97bdfa-8926-4c5c-9924-d06d84b68cbe-kube-api-access-zwsw7\") pod \"payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c\" (UID: \"ec97bdfa-8926-4c5c-9924-d06d84b68cbe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"
Sep 08 11:43:25.929333 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:25.929304    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"
Sep 08 11:43:26.044234 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:26.044145    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:26.109596 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:26.109555    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:26.206739 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:26.206716    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"]
Sep 08 11:43:26.208225 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:43:26.208199    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec97bdfa_8926_4c5c_9924_d06d84b68cbe.slice/crio-0d1635dfb0c8942b59530508f3e16c22e78a0f46f88b132f9628644da78282ed WatchSource:0}: Error finding container 0d1635dfb0c8942b59530508f3e16c22e78a0f46f88b132f9628644da78282ed: Status 404 returned error can't find the container with id 0d1635dfb0c8942b59530508f3e16c22e78a0f46f88b132f9628644da78282ed
Sep 08 11:43:26.235169 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:26.235115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c" event={"ID":"ec97bdfa-8926-4c5c-9924-d06d84b68cbe","Type":"ContainerStarted","Data":"0d1635dfb0c8942b59530508f3e16c22e78a0f46f88b132f9628644da78282ed"}
Sep 08 11:43:26.465407 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:26.465370    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:27.243692 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:27.243649    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c" event={"ID":"ec97bdfa-8926-4c5c-9924-d06d84b68cbe","Type":"ContainerStarted","Data":"771152659b9a61d8318e3fa35ccb13161c6315615c0bf5ce90e275f91b500e9d"}
Sep 08 11:43:27.244160 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:27.243779    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"
Sep 08 11:43:27.279185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:27.279130    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c" podStartSLOduration=2.279112167 podStartE2EDuration="2.279112167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:43:27.277489469 +0000 UTC m=+813.844250003" watchObservedRunningTime="2026-09-08 11:43:27.279112167 +0000 UTC m=+813.845872710"
Sep 08 11:43:28.336887 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:28.336853    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5"]
Sep 08 11:43:28.340799 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:28.340779    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5"
Sep 08 11:43:28.343477 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:28.343441    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:43:28.343579 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:28.343445    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:43:28.349497 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:28.349456    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5"]
Sep 08 11:43:28.488850 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:28.488810    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8869507-a2cd-41a5-9600-1d5184c8ed14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5\" (UID: \"f8869507-a2cd-41a5-9600-1d5184c8ed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5"
Sep 08 11:43:28.589559 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:28.589448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8869507-a2cd-41a5-9600-1d5184c8ed14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5\" (UID: \"f8869507-a2cd-41a5-9600-1d5184c8ed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5"
Sep 08 11:43:28.590082 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:28.590061    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8869507-a2cd-41a5-9600-1d5184c8ed14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5\" (UID: \"f8869507-a2cd-41a5-9600-1d5184c8ed14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5"
Sep 08 11:43:28.652121 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:28.652082    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5"
Sep 08 11:43:28.928922 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:28.928899    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5"]
Sep 08 11:43:28.931268 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:43:28.931237    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8869507_a2cd_41a5_9600_1d5184c8ed14.slice/crio-a3cb1df379478c21f5dc0ca3ab3879df873be0f1298f7cb34675c1ba04d7f12e WatchSource:0}: Error finding container a3cb1df379478c21f5dc0ca3ab3879df873be0f1298f7cb34675c1ba04d7f12e: Status 404 returned error can't find the container with id a3cb1df379478c21f5dc0ca3ab3879df873be0f1298f7cb34675c1ba04d7f12e
Sep 08 11:43:29.253670 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:29.253632    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5" event={"ID":"f8869507-a2cd-41a5-9600-1d5184c8ed14","Type":"ContainerStarted","Data":"a3cb1df379478c21f5dc0ca3ab3879df873be0f1298f7cb34675c1ba04d7f12e"}
Sep 08 11:43:31.026174 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:31.026141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:31.864606 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:31.864515    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:32.271089 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:32.271049    2561 generic.go:356] "Generic (PLEG): container finished" podID="f8869507-a2cd-41a5-9600-1d5184c8ed14" containerID="5acbdb09b425558986af66ae33706e0b3fa2196e02c2b6a1ca20f44701579a6a" exitCode=0
Sep 08 11:43:32.271538 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:32.271140    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5" event={"ID":"f8869507-a2cd-41a5-9600-1d5184c8ed14","Type":"ContainerDied","Data":"5acbdb09b425558986af66ae33706e0b3fa2196e02c2b6a1ca20f44701579a6a"}
Sep 08 11:43:33.403622 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:33.403600    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5"
Sep 08 11:43:33.468402 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:33.468366    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:33.534708 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:33.534633    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8869507-a2cd-41a5-9600-1d5184c8ed14-maas-api-service-ca\") pod \"f8869507-a2cd-41a5-9600-1d5184c8ed14\" (UID: \"f8869507-a2cd-41a5-9600-1d5184c8ed14\") "
Sep 08 11:43:33.534984 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:33.534958    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8869507-a2cd-41a5-9600-1d5184c8ed14-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8869507-a2cd-41a5-9600-1d5184c8ed14" (UID: "f8869507-a2cd-41a5-9600-1d5184c8ed14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:43:33.572664 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:33.572626    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:33.635398 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:33.635364    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8869507-a2cd-41a5-9600-1d5184c8ed14-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:43:34.283431 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:34.283358    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5"
Sep 08 11:43:34.283591 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:34.283355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5" event={"ID":"f8869507-a2cd-41a5-9600-1d5184c8ed14","Type":"ContainerDied","Data":"a3cb1df379478c21f5dc0ca3ab3879df873be0f1298f7cb34675c1ba04d7f12e"}
Sep 08 11:43:34.283591 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:34.283502    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3cb1df379478c21f5dc0ca3ab3879df873be0f1298f7cb34675c1ba04d7f12e"
Sep 08 11:43:34.385652 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:34.385606    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:38.250826 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:38.250792    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"
Sep 08 11:43:44.132722 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:44.132670    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:44.861189 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:44.861158    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:44.885026 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:44.884996    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"]
Sep 08 11:43:44.885252 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:44.885221    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z" podUID="041e8061-0ede-4a51-94a7-10be919ba3be" containerName="payload-processing" containerID="cri-o://64c1417e6b88d2716db7c0863c75f01e946b00305229e010c099956ad2d3ea22" gracePeriod=30
Sep 08 11:43:44.894829 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:44.894801    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"]
Sep 08 11:43:44.895101 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:44.895009    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc" podUID="6348aba7-6439-4400-80ef-29ac41a2bac6" containerName="payload-pre-processing" containerID="cri-o://455cdb078f984bd09c0fc5a80c13ca29d028ee799d1d22f3c818527595df5c36" gracePeriod=30
Sep 08 11:43:44.935039 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:44.935012    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:45.004753 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.004711    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:45.071095 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.071031    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5"]
Sep 08 11:43:45.076355 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.076329    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7d3a6048-pxdw5"]
Sep 08 11:43:45.078915 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.078888    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:45.165258 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.165234    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"
Sep 08 11:43:45.168522 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.168500    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"
Sep 08 11:43:45.335661 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.335631    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6348aba7-6439-4400-80ef-29ac41a2bac6-plugins-config-volume\") pod \"6348aba7-6439-4400-80ef-29ac41a2bac6\" (UID: \"6348aba7-6439-4400-80ef-29ac41a2bac6\") "
Sep 08 11:43:45.335821 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.335680    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp2vf\" (UniqueName: \"kubernetes.io/projected/041e8061-0ede-4a51-94a7-10be919ba3be-kube-api-access-kp2vf\") pod \"041e8061-0ede-4a51-94a7-10be919ba3be\" (UID: \"041e8061-0ede-4a51-94a7-10be919ba3be\") "
Sep 08 11:43:45.335821 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.335706    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041e8061-0ede-4a51-94a7-10be919ba3be-plugins-config-volume\") pod \"041e8061-0ede-4a51-94a7-10be919ba3be\" (UID: \"041e8061-0ede-4a51-94a7-10be919ba3be\") "
Sep 08 11:43:45.335821 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.335771    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxrpn\" (UniqueName: \"kubernetes.io/projected/6348aba7-6439-4400-80ef-29ac41a2bac6-kube-api-access-mxrpn\") pod \"6348aba7-6439-4400-80ef-29ac41a2bac6\" (UID: \"6348aba7-6439-4400-80ef-29ac41a2bac6\") "
Sep 08 11:43:45.336079 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.336053    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6348aba7-6439-4400-80ef-29ac41a2bac6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6348aba7-6439-4400-80ef-29ac41a2bac6" (UID: "6348aba7-6439-4400-80ef-29ac41a2bac6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:43:45.336144 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.336098    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041e8061-0ede-4a51-94a7-10be919ba3be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "041e8061-0ede-4a51-94a7-10be919ba3be" (UID: "041e8061-0ede-4a51-94a7-10be919ba3be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:43:45.337903 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.337878    2561 generic.go:356] "Generic (PLEG): container finished" podID="041e8061-0ede-4a51-94a7-10be919ba3be" containerID="64c1417e6b88d2716db7c0863c75f01e946b00305229e010c099956ad2d3ea22" exitCode=0
Sep 08 11:43:45.338020 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.337931    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"
Sep 08 11:43:45.338020 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.337963    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z" event={"ID":"041e8061-0ede-4a51-94a7-10be919ba3be","Type":"ContainerDied","Data":"64c1417e6b88d2716db7c0863c75f01e946b00305229e010c099956ad2d3ea22"}
Sep 08 11:43:45.338170 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.338048    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z" event={"ID":"041e8061-0ede-4a51-94a7-10be919ba3be","Type":"ContainerDied","Data":"3b12c0dc89d10a2cbc39d48ca3bdb4ca5687a08cc313924056c08c0e6641b2aa"}
Sep 08 11:43:45.338170 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.338047    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041e8061-0ede-4a51-94a7-10be919ba3be-kube-api-access-kp2vf" (OuterVolumeSpecName: "kube-api-access-kp2vf") pod "041e8061-0ede-4a51-94a7-10be919ba3be" (UID: "041e8061-0ede-4a51-94a7-10be919ba3be"). InnerVolumeSpecName "kube-api-access-kp2vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:43:45.338170 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.338081    2561 scope.go:117] "RemoveContainer" containerID="64c1417e6b88d2716db7c0863c75f01e946b00305229e010c099956ad2d3ea22"
Sep 08 11:43:45.338170 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.338071    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6348aba7-6439-4400-80ef-29ac41a2bac6-kube-api-access-mxrpn" (OuterVolumeSpecName: "kube-api-access-mxrpn") pod "6348aba7-6439-4400-80ef-29ac41a2bac6" (UID: "6348aba7-6439-4400-80ef-29ac41a2bac6"). InnerVolumeSpecName "kube-api-access-mxrpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:43:45.339254 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.339234    2561 generic.go:356] "Generic (PLEG): container finished" podID="6348aba7-6439-4400-80ef-29ac41a2bac6" containerID="455cdb078f984bd09c0fc5a80c13ca29d028ee799d1d22f3c818527595df5c36" exitCode=0
Sep 08 11:43:45.339334 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.339265    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc" event={"ID":"6348aba7-6439-4400-80ef-29ac41a2bac6","Type":"ContainerDied","Data":"455cdb078f984bd09c0fc5a80c13ca29d028ee799d1d22f3c818527595df5c36"}
Sep 08 11:43:45.339334 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.339292    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"
Sep 08 11:43:45.339334 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.339304    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc" event={"ID":"6348aba7-6439-4400-80ef-29ac41a2bac6","Type":"ContainerDied","Data":"1ffeb873e20cdf3fdb114c64229a6802a72c8da05ae2b91f017db528594b1034"}
Sep 08 11:43:45.346286 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.346267    2561 scope.go:117] "RemoveContainer" containerID="64c1417e6b88d2716db7c0863c75f01e946b00305229e010c099956ad2d3ea22"
Sep 08 11:43:45.346659 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:43:45.346573    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c1417e6b88d2716db7c0863c75f01e946b00305229e010c099956ad2d3ea22\": container with ID starting with 64c1417e6b88d2716db7c0863c75f01e946b00305229e010c099956ad2d3ea22 not found: ID does not exist" containerID="64c1417e6b88d2716db7c0863c75f01e946b00305229e010c099956ad2d3ea22"
Sep 08 11:43:45.346659 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.346606    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c1417e6b88d2716db7c0863c75f01e946b00305229e010c099956ad2d3ea22"} err="failed to get container status \"64c1417e6b88d2716db7c0863c75f01e946b00305229e010c099956ad2d3ea22\": rpc error: code = NotFound desc = could not find container \"64c1417e6b88d2716db7c0863c75f01e946b00305229e010c099956ad2d3ea22\": container with ID starting with 64c1417e6b88d2716db7c0863c75f01e946b00305229e010c099956ad2d3ea22 not found: ID does not exist"
Sep 08 11:43:45.346659 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.346628    2561 scope.go:117] "RemoveContainer" containerID="455cdb078f984bd09c0fc5a80c13ca29d028ee799d1d22f3c818527595df5c36"
Sep 08 11:43:45.354078 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.354060    2561 scope.go:117] "RemoveContainer" containerID="455cdb078f984bd09c0fc5a80c13ca29d028ee799d1d22f3c818527595df5c36"
Sep 08 11:43:45.354332 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:43:45.354314    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455cdb078f984bd09c0fc5a80c13ca29d028ee799d1d22f3c818527595df5c36\": container with ID starting with 455cdb078f984bd09c0fc5a80c13ca29d028ee799d1d22f3c818527595df5c36 not found: ID does not exist" containerID="455cdb078f984bd09c0fc5a80c13ca29d028ee799d1d22f3c818527595df5c36"
Sep 08 11:43:45.354379 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.354339    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455cdb078f984bd09c0fc5a80c13ca29d028ee799d1d22f3c818527595df5c36"} err="failed to get container status \"455cdb078f984bd09c0fc5a80c13ca29d028ee799d1d22f3c818527595df5c36\": rpc error: code = NotFound desc = could not find container \"455cdb078f984bd09c0fc5a80c13ca29d028ee799d1d22f3c818527595df5c36\": container with ID starting with 455cdb078f984bd09c0fc5a80c13ca29d028ee799d1d22f3c818527595df5c36 not found: ID does not exist"
Sep 08 11:43:45.362164 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.362143    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"]
Sep 08 11:43:45.367088 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.367068    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7d3a6048-78b9fbf858-p75vc"]
Sep 08 11:43:45.436516 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.436443    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxrpn\" (UniqueName: \"kubernetes.io/projected/6348aba7-6439-4400-80ef-29ac41a2bac6-kube-api-access-mxrpn\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:43:45.436516 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.436503    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6348aba7-6439-4400-80ef-29ac41a2bac6-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:43:45.436656 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.436520    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp2vf\" (UniqueName: \"kubernetes.io/projected/041e8061-0ede-4a51-94a7-10be919ba3be-kube-api-access-kp2vf\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:43:45.436656 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.436532    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041e8061-0ede-4a51-94a7-10be919ba3be-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:43:45.666745 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.666709    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"]
Sep 08 11:43:45.672026 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.671999    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d3a6048-5dfbb4c477-l2l7z"]
Sep 08 11:43:45.960657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.960625    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="041e8061-0ede-4a51-94a7-10be919ba3be" path="/var/lib/kubelet/pods/041e8061-0ede-4a51-94a7-10be919ba3be/volumes"
Sep 08 11:43:45.961061 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.961041    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6348aba7-6439-4400-80ef-29ac41a2bac6" path="/var/lib/kubelet/pods/6348aba7-6439-4400-80ef-29ac41a2bac6/volumes"
Sep 08 11:43:45.961392 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:45.961374    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8869507-a2cd-41a5-9600-1d5184c8ed14" path="/var/lib/kubelet/pods/f8869507-a2cd-41a5-9600-1d5184c8ed14/volumes"
Sep 08 11:43:46.906923 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:46.906886    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:47.002975 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:47.002945    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:47.988360 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:47.988323    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:48.479433 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:48.479403    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:49.195028 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:49.194995    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:50.018607 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.018572    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"]
Sep 08 11:43:50.018926 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.018914    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041e8061-0ede-4a51-94a7-10be919ba3be" containerName="payload-processing"
Sep 08 11:43:50.018975 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.018927    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="041e8061-0ede-4a51-94a7-10be919ba3be" containerName="payload-processing"
Sep 08 11:43:50.018975 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.018935    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8869507-a2cd-41a5-9600-1d5184c8ed14" containerName="revoke-keys"
Sep 08 11:43:50.018975 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.018941    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8869507-a2cd-41a5-9600-1d5184c8ed14" containerName="revoke-keys"
Sep 08 11:43:50.018975 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.018952    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6348aba7-6439-4400-80ef-29ac41a2bac6" containerName="payload-pre-processing"
Sep 08 11:43:50.018975 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.018957    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6348aba7-6439-4400-80ef-29ac41a2bac6" containerName="payload-pre-processing"
Sep 08 11:43:50.019131 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.019018    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8869507-a2cd-41a5-9600-1d5184c8ed14" containerName="revoke-keys"
Sep 08 11:43:50.019131 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.019026    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6348aba7-6439-4400-80ef-29ac41a2bac6" containerName="payload-pre-processing"
Sep 08 11:43:50.019131 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.019035    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="041e8061-0ede-4a51-94a7-10be919ba3be" containerName="payload-processing"
Sep 08 11:43:50.021902 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.021882    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"
Sep 08 11:43:50.024417 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.024398    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ac040d69\""
Sep 08 11:43:50.024542 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.024402    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ac040d69-dockercfg-sj8qv\""
Sep 08 11:43:50.033282 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.033260    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"]
Sep 08 11:43:50.067520 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.067493    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"]
Sep 08 11:43:50.070723 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.070706    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"
Sep 08 11:43:50.081711 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.081691    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"]
Sep 08 11:43:50.176619 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.176577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a37bbe1-49ef-4e38-8be6-e2660ab483dc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k\" (UID: \"8a37bbe1-49ef-4e38-8be6-e2660ab483dc\") " pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"
Sep 08 11:43:50.176770 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.176643    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r45d\" (UniqueName: \"kubernetes.io/projected/8a37bbe1-49ef-4e38-8be6-e2660ab483dc-kube-api-access-4r45d\") pod \"payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k\" (UID: \"8a37bbe1-49ef-4e38-8be6-e2660ab483dc\") " pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"
Sep 08 11:43:50.176770 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.176690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqtkf\" (UniqueName: \"kubernetes.io/projected/a72fddc6-0ad2-4eab-9753-28eb08aafee1-kube-api-access-vqtkf\") pod \"payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7\" (UID: \"a72fddc6-0ad2-4eab-9753-28eb08aafee1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"
Sep 08 11:43:50.176770 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.176723    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72fddc6-0ad2-4eab-9753-28eb08aafee1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7\" (UID: \"a72fddc6-0ad2-4eab-9753-28eb08aafee1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"
Sep 08 11:43:50.277277 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.277189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r45d\" (UniqueName: \"kubernetes.io/projected/8a37bbe1-49ef-4e38-8be6-e2660ab483dc-kube-api-access-4r45d\") pod \"payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k\" (UID: \"8a37bbe1-49ef-4e38-8be6-e2660ab483dc\") " pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"
Sep 08 11:43:50.277277 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.277237    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqtkf\" (UniqueName: \"kubernetes.io/projected/a72fddc6-0ad2-4eab-9753-28eb08aafee1-kube-api-access-vqtkf\") pod \"payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7\" (UID: \"a72fddc6-0ad2-4eab-9753-28eb08aafee1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"
Sep 08 11:43:50.277849 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.277359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72fddc6-0ad2-4eab-9753-28eb08aafee1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7\" (UID: \"a72fddc6-0ad2-4eab-9753-28eb08aafee1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"
Sep 08 11:43:50.277849 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.277423    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a37bbe1-49ef-4e38-8be6-e2660ab483dc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k\" (UID: \"8a37bbe1-49ef-4e38-8be6-e2660ab483dc\") " pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"
Sep 08 11:43:50.278164 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.278141    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72fddc6-0ad2-4eab-9753-28eb08aafee1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7\" (UID: \"a72fddc6-0ad2-4eab-9753-28eb08aafee1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"
Sep 08 11:43:50.278250 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.278230    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a37bbe1-49ef-4e38-8be6-e2660ab483dc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k\" (UID: \"8a37bbe1-49ef-4e38-8be6-e2660ab483dc\") " pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"
Sep 08 11:43:50.294690 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.294666    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqtkf\" (UniqueName: \"kubernetes.io/projected/a72fddc6-0ad2-4eab-9753-28eb08aafee1-kube-api-access-vqtkf\") pod \"payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7\" (UID: \"a72fddc6-0ad2-4eab-9753-28eb08aafee1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"
Sep 08 11:43:50.299296 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.299277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r45d\" (UniqueName: \"kubernetes.io/projected/8a37bbe1-49ef-4e38-8be6-e2660ab483dc-kube-api-access-4r45d\") pod \"payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k\" (UID: \"8a37bbe1-49ef-4e38-8be6-e2660ab483dc\") " pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"
Sep 08 11:43:50.332279 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.332258    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"
Sep 08 11:43:50.380678 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.380652    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"
Sep 08 11:43:50.610701 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.610679    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"]
Sep 08 11:43:50.612397 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:43:50.612370    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a37bbe1_49ef_4e38_8be6_e2660ab483dc.slice/crio-62754c0bc899e10513ec55aff0a6f59eddefa66fcb91749648443112d849e159 WatchSource:0}: Error finding container 62754c0bc899e10513ec55aff0a6f59eddefa66fcb91749648443112d849e159: Status 404 returned error can't find the container with id 62754c0bc899e10513ec55aff0a6f59eddefa66fcb91749648443112d849e159
Sep 08 11:43:50.663662 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:50.663638    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"]
Sep 08 11:43:50.665538 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:43:50.665513    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72fddc6_0ad2_4eab_9753_28eb08aafee1.slice/crio-9a875fe35e9a73533784e9a6fa7998ea112f03fae78f60df24a707948c990bb6 WatchSource:0}: Error finding container 9a875fe35e9a73533784e9a6fa7998ea112f03fae78f60df24a707948c990bb6: Status 404 returned error can't find the container with id 9a875fe35e9a73533784e9a6fa7998ea112f03fae78f60df24a707948c990bb6
Sep 08 11:43:51.101502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:51.101441    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:51.170327 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:51.170295    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:43:51.374005 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:51.373896    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7" event={"ID":"a72fddc6-0ad2-4eab-9753-28eb08aafee1","Type":"ContainerStarted","Data":"601c688afc08091b786540cbe8566a3d8d09f409f1d414c271bd673f907493eb"}
Sep 08 11:43:51.374005 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:51.373997    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7" event={"ID":"a72fddc6-0ad2-4eab-9753-28eb08aafee1","Type":"ContainerStarted","Data":"9a875fe35e9a73533784e9a6fa7998ea112f03fae78f60df24a707948c990bb6"}
Sep 08 11:43:51.374517 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:51.374026    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"
Sep 08 11:43:51.375664 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:51.375637    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k" event={"ID":"8a37bbe1-49ef-4e38-8be6-e2660ab483dc","Type":"ContainerStarted","Data":"41f2fb64a54a49b74f624988a5aad2192f592f7119feb3bab44a4ef04eceb051"}
Sep 08 11:43:51.375786 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:51.375684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k" event={"ID":"8a37bbe1-49ef-4e38-8be6-e2660ab483dc","Type":"ContainerStarted","Data":"62754c0bc899e10513ec55aff0a6f59eddefa66fcb91749648443112d849e159"}
Sep 08 11:43:51.375786 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:51.375764    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"
Sep 08 11:43:51.392180 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:51.392111    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7" podStartSLOduration=1.392098052 podStartE2EDuration="1.392098052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:43:51.391337749 +0000 UTC m=+837.958098287" watchObservedRunningTime="2026-09-08 11:43:51.392098052 +0000 UTC m=+837.958858593"
Sep 08 11:43:51.409360 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:43:51.409310    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k" podStartSLOduration=2.409294397 podStartE2EDuration="2.409294397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:43:51.408775254 +0000 UTC m=+837.975535799" watchObservedRunningTime="2026-09-08 11:43:51.409294397 +0000 UTC m=+837.976054938"
Sep 08 11:44:02.383150 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:02.383117    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"
Sep 08 11:44:02.383650 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:02.383274    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"
Sep 08 11:44:14.156267 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:14.156232    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7"]
Sep 08 11:44:14.161938 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:14.161920    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7"
Sep 08 11:44:14.164456 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:14.164434    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:44:14.164594 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:14.164479    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:44:14.169852 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:14.169774    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7"]
Sep 08 11:44:14.247633 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:14.247598    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32722d39-8789-4fd4-84eb-0012e9ee95c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7\" (UID: \"32722d39-8789-4fd4-84eb-0012e9ee95c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7"
Sep 08 11:44:14.348313 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:14.348273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32722d39-8789-4fd4-84eb-0012e9ee95c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7\" (UID: \"32722d39-8789-4fd4-84eb-0012e9ee95c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7"
Sep 08 11:44:14.348905 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:14.348885    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32722d39-8789-4fd4-84eb-0012e9ee95c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7\" (UID: \"32722d39-8789-4fd4-84eb-0012e9ee95c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7"
Sep 08 11:44:14.474370 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:14.474336    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7"
Sep 08 11:44:14.743746 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:14.743721    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7"]
Sep 08 11:44:14.745906 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:44:14.745877    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32722d39_8789_4fd4_84eb_0012e9ee95c0.slice/crio-c89c866dd6129facf900b8463c8af951f4bfb95c0ee3ed09d990779f2dcee518 WatchSource:0}: Error finding container c89c866dd6129facf900b8463c8af951f4bfb95c0ee3ed09d990779f2dcee518: Status 404 returned error can't find the container with id c89c866dd6129facf900b8463c8af951f4bfb95c0ee3ed09d990779f2dcee518
Sep 08 11:44:15.485124 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:15.485093    2561 generic.go:356] "Generic (PLEG): container finished" podID="32722d39-8789-4fd4-84eb-0012e9ee95c0" containerID="8b446ad5f83786582758f384f073cc6fcba4bfa381ba17135cef189f3045f86e" exitCode=0
Sep 08 11:44:15.485487 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:15.485173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7" event={"ID":"32722d39-8789-4fd4-84eb-0012e9ee95c0","Type":"ContainerDied","Data":"8b446ad5f83786582758f384f073cc6fcba4bfa381ba17135cef189f3045f86e"}
Sep 08 11:44:15.485487 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:15.485217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7" event={"ID":"32722d39-8789-4fd4-84eb-0012e9ee95c0","Type":"ContainerStarted","Data":"c89c866dd6129facf900b8463c8af951f4bfb95c0ee3ed09d990779f2dcee518"}
Sep 08 11:44:15.848895 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:15.848863    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:16.620316 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:16.620291    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7"
Sep 08 11:44:16.663605 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:16.663568    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32722d39-8789-4fd4-84eb-0012e9ee95c0-maas-api-service-ca\") pod \"32722d39-8789-4fd4-84eb-0012e9ee95c0\" (UID: \"32722d39-8789-4fd4-84eb-0012e9ee95c0\") "
Sep 08 11:44:16.663900 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:16.663878    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32722d39-8789-4fd4-84eb-0012e9ee95c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "32722d39-8789-4fd4-84eb-0012e9ee95c0" (UID: "32722d39-8789-4fd4-84eb-0012e9ee95c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:44:16.764302 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:16.764219    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32722d39-8789-4fd4-84eb-0012e9ee95c0-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:44:17.496328 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:17.496294    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7" event={"ID":"32722d39-8789-4fd4-84eb-0012e9ee95c0","Type":"ContainerDied","Data":"c89c866dd6129facf900b8463c8af951f4bfb95c0ee3ed09d990779f2dcee518"}
Sep 08 11:44:17.496328 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:17.496318    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7"
Sep 08 11:44:17.496571 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:17.496350    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c89c866dd6129facf900b8463c8af951f4bfb95c0ee3ed09d990779f2dcee518"
Sep 08 11:44:20.194767 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.194738    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:20.231219 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.231187    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"]
Sep 08 11:44:20.231444 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.231415    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k" podUID="8a37bbe1-49ef-4e38-8be6-e2660ab483dc" containerName="payload-processing" containerID="cri-o://41f2fb64a54a49b74f624988a5aad2192f592f7119feb3bab44a4ef04eceb051" gracePeriod=30
Sep 08 11:44:20.241887 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.241860    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"]
Sep 08 11:44:20.242089 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.242059    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7" podUID="a72fddc6-0ad2-4eab-9753-28eb08aafee1" containerName="payload-pre-processing" containerID="cri-o://601c688afc08091b786540cbe8566a3d8d09f409f1d414c271bd673f907493eb" gracePeriod=30
Sep 08 11:44:20.289014 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.288984    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:20.361189 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.361157    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:20.459028 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.458964    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:20.480742 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.480710    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7"]
Sep 08 11:44:20.484813 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.484786    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ac040d69-8hpz7"]
Sep 08 11:44:20.512243 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.512218    2561 generic.go:356] "Generic (PLEG): container finished" podID="a72fddc6-0ad2-4eab-9753-28eb08aafee1" containerID="601c688afc08091b786540cbe8566a3d8d09f409f1d414c271bd673f907493eb" exitCode=0
Sep 08 11:44:20.512360 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.512307    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7" event={"ID":"a72fddc6-0ad2-4eab-9753-28eb08aafee1","Type":"ContainerDied","Data":"601c688afc08091b786540cbe8566a3d8d09f409f1d414c271bd673f907493eb"}
Sep 08 11:44:20.512498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.512361    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7" event={"ID":"a72fddc6-0ad2-4eab-9753-28eb08aafee1","Type":"ContainerDied","Data":"9a875fe35e9a73533784e9a6fa7998ea112f03fae78f60df24a707948c990bb6"}
Sep 08 11:44:20.512498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.512387    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a875fe35e9a73533784e9a6fa7998ea112f03fae78f60df24a707948c990bb6"
Sep 08 11:44:20.513740 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.513720    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a37bbe1-49ef-4e38-8be6-e2660ab483dc" containerID="41f2fb64a54a49b74f624988a5aad2192f592f7119feb3bab44a4ef04eceb051" exitCode=0
Sep 08 11:44:20.513848 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.513780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k" event={"ID":"8a37bbe1-49ef-4e38-8be6-e2660ab483dc","Type":"ContainerDied","Data":"41f2fb64a54a49b74f624988a5aad2192f592f7119feb3bab44a4ef04eceb051"}
Sep 08 11:44:20.513848 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.513822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k" event={"ID":"8a37bbe1-49ef-4e38-8be6-e2660ab483dc","Type":"ContainerDied","Data":"62754c0bc899e10513ec55aff0a6f59eddefa66fcb91749648443112d849e159"}
Sep 08 11:44:20.513949 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.513847    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62754c0bc899e10513ec55aff0a6f59eddefa66fcb91749648443112d849e159"
Sep 08 11:44:20.514580 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.514562    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"
Sep 08 11:44:20.517854 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.517836    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"
Sep 08 11:44:20.534924 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.534904    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:20.589648 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.589606    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a37bbe1-49ef-4e38-8be6-e2660ab483dc-plugins-config-volume\") pod \"8a37bbe1-49ef-4e38-8be6-e2660ab483dc\" (UID: \"8a37bbe1-49ef-4e38-8be6-e2660ab483dc\") "
Sep 08 11:44:20.589839 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.589681    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqtkf\" (UniqueName: \"kubernetes.io/projected/a72fddc6-0ad2-4eab-9753-28eb08aafee1-kube-api-access-vqtkf\") pod \"a72fddc6-0ad2-4eab-9753-28eb08aafee1\" (UID: \"a72fddc6-0ad2-4eab-9753-28eb08aafee1\") "
Sep 08 11:44:20.589839 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.589719    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r45d\" (UniqueName: \"kubernetes.io/projected/8a37bbe1-49ef-4e38-8be6-e2660ab483dc-kube-api-access-4r45d\") pod \"8a37bbe1-49ef-4e38-8be6-e2660ab483dc\" (UID: \"8a37bbe1-49ef-4e38-8be6-e2660ab483dc\") "
Sep 08 11:44:20.589839 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.589754    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72fddc6-0ad2-4eab-9753-28eb08aafee1-plugins-config-volume\") pod \"a72fddc6-0ad2-4eab-9753-28eb08aafee1\" (UID: \"a72fddc6-0ad2-4eab-9753-28eb08aafee1\") "
Sep 08 11:44:20.590108 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.590079    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a37bbe1-49ef-4e38-8be6-e2660ab483dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a37bbe1-49ef-4e38-8be6-e2660ab483dc" (UID: "8a37bbe1-49ef-4e38-8be6-e2660ab483dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:44:20.590365 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.590343    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72fddc6-0ad2-4eab-9753-28eb08aafee1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a72fddc6-0ad2-4eab-9753-28eb08aafee1" (UID: "a72fddc6-0ad2-4eab-9753-28eb08aafee1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:44:20.592590 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.592568    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72fddc6-0ad2-4eab-9753-28eb08aafee1-kube-api-access-vqtkf" (OuterVolumeSpecName: "kube-api-access-vqtkf") pod "a72fddc6-0ad2-4eab-9753-28eb08aafee1" (UID: "a72fddc6-0ad2-4eab-9753-28eb08aafee1"). InnerVolumeSpecName "kube-api-access-vqtkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:44:20.592719 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.592663    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a37bbe1-49ef-4e38-8be6-e2660ab483dc-kube-api-access-4r45d" (OuterVolumeSpecName: "kube-api-access-4r45d") pod "8a37bbe1-49ef-4e38-8be6-e2660ab483dc" (UID: "8a37bbe1-49ef-4e38-8be6-e2660ab483dc"). InnerVolumeSpecName "kube-api-access-4r45d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:44:20.645659 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.645633    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:20.690743 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.690709    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a37bbe1-49ef-4e38-8be6-e2660ab483dc-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:44:20.690743 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.690739    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqtkf\" (UniqueName: \"kubernetes.io/projected/a72fddc6-0ad2-4eab-9753-28eb08aafee1-kube-api-access-vqtkf\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:44:20.690924 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.690754    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r45d\" (UniqueName: \"kubernetes.io/projected/8a37bbe1-49ef-4e38-8be6-e2660ab483dc-kube-api-access-4r45d\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:44:20.690924 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:20.690770    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a72fddc6-0ad2-4eab-9753-28eb08aafee1-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:44:21.083532 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:21.083497    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:21.519393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:21.519361    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"
Sep 08 11:44:21.519393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:21.519375    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"
Sep 08 11:44:21.545300 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:21.545266    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"]
Sep 08 11:44:21.552827 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:21.552799    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ac040d69-85f5fb9f58-dt47k"]
Sep 08 11:44:21.566635 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:21.566603    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"]
Sep 08 11:44:21.573895 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:21.573873    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ac040d69-7cfffdf7cd-4wlk7"]
Sep 08 11:44:21.960611 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:21.960579    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32722d39-8789-4fd4-84eb-0012e9ee95c0" path="/var/lib/kubelet/pods/32722d39-8789-4fd4-84eb-0012e9ee95c0/volumes"
Sep 08 11:44:21.960932 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:21.960920    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a37bbe1-49ef-4e38-8be6-e2660ab483dc" path="/var/lib/kubelet/pods/8a37bbe1-49ef-4e38-8be6-e2660ab483dc/volumes"
Sep 08 11:44:21.961233 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:21.961218    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a72fddc6-0ad2-4eab-9753-28eb08aafee1" path="/var/lib/kubelet/pods/a72fddc6-0ad2-4eab-9753-28eb08aafee1/volumes"
Sep 08 11:44:22.322750 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:22.322671    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:22.422394 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:22.422353    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:24.361826 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:24.361793    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:24.455220 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:24.455189    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:25.545684 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:25.545646    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:25.647039 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:25.647006    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:25.899653 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:25.899545    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:26.056490 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.056422    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:26.154980 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.154899    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:26.173883 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.173830    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"]
Sep 08 11:44:26.174174 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.174163    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a72fddc6-0ad2-4eab-9753-28eb08aafee1" containerName="payload-pre-processing"
Sep 08 11:44:26.174218 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.174176    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a72fddc6-0ad2-4eab-9753-28eb08aafee1" containerName="payload-pre-processing"
Sep 08 11:44:26.174218 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.174191    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a37bbe1-49ef-4e38-8be6-e2660ab483dc" containerName="payload-processing"
Sep 08 11:44:26.174218 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.174196    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a37bbe1-49ef-4e38-8be6-e2660ab483dc" containerName="payload-processing"
Sep 08 11:44:26.174218 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.174205    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32722d39-8789-4fd4-84eb-0012e9ee95c0" containerName="revoke-keys"
Sep 08 11:44:26.174218 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.174211    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="32722d39-8789-4fd4-84eb-0012e9ee95c0" containerName="revoke-keys"
Sep 08 11:44:26.174367 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.174264    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a37bbe1-49ef-4e38-8be6-e2660ab483dc" containerName="payload-processing"
Sep 08 11:44:26.174367 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.174276    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a72fddc6-0ad2-4eab-9753-28eb08aafee1" containerName="payload-pre-processing"
Sep 08 11:44:26.174367 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.174282    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="32722d39-8789-4fd4-84eb-0012e9ee95c0" containerName="revoke-keys"
Sep 08 11:44:26.178590 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.178572    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"
Sep 08 11:44:26.181031 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.181016    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b16466d7-dockercfg-jnk9j\""
Sep 08 11:44:26.181272 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.181252    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b16466d7\""
Sep 08 11:44:26.190598 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.190576    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"]
Sep 08 11:44:26.238740 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.238708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbgvp\" (UniqueName: \"kubernetes.io/projected/9e9fabd2-88a8-4522-8950-6a4d2248a5bd-kube-api-access-mbgvp\") pod \"payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb\" (UID: \"9e9fabd2-88a8-4522-8950-6a4d2248a5bd\") " pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"
Sep 08 11:44:26.238905 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.238756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9fabd2-88a8-4522-8950-6a4d2248a5bd-plugins-config-volume\") pod \"payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb\" (UID: \"9e9fabd2-88a8-4522-8950-6a4d2248a5bd\") " pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"
Sep 08 11:44:26.339432 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.339396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9fabd2-88a8-4522-8950-6a4d2248a5bd-plugins-config-volume\") pod \"payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb\" (UID: \"9e9fabd2-88a8-4522-8950-6a4d2248a5bd\") " pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"
Sep 08 11:44:26.339599 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.339576    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbgvp\" (UniqueName: \"kubernetes.io/projected/9e9fabd2-88a8-4522-8950-6a4d2248a5bd-kube-api-access-mbgvp\") pod \"payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb\" (UID: \"9e9fabd2-88a8-4522-8950-6a4d2248a5bd\") " pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"
Sep 08 11:44:26.340109 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.340088    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9fabd2-88a8-4522-8950-6a4d2248a5bd-plugins-config-volume\") pod \"payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb\" (UID: \"9e9fabd2-88a8-4522-8950-6a4d2248a5bd\") " pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"
Sep 08 11:44:26.349788 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.349764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbgvp\" (UniqueName: \"kubernetes.io/projected/9e9fabd2-88a8-4522-8950-6a4d2248a5bd-kube-api-access-mbgvp\") pod \"payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb\" (UID: \"9e9fabd2-88a8-4522-8950-6a4d2248a5bd\") " pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"
Sep 08 11:44:26.490103 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.490069    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"
Sep 08 11:44:26.763841 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:26.763783    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"]
Sep 08 11:44:26.766070 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:44:26.766036    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9fabd2_88a8_4522_8950_6a4d2248a5bd.slice/crio-063689a9f6deca8ec04097e06719d8ed9b01ce2f8b4b90acbc34408952f12141 WatchSource:0}: Error finding container 063689a9f6deca8ec04097e06719d8ed9b01ce2f8b4b90acbc34408952f12141: Status 404 returned error can't find the container with id 063689a9f6deca8ec04097e06719d8ed9b01ce2f8b4b90acbc34408952f12141
Sep 08 11:44:27.550902 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:27.550871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb" event={"ID":"9e9fabd2-88a8-4522-8950-6a4d2248a5bd","Type":"ContainerStarted","Data":"1133e44f548f2a36c286c88987aabb7ff5ef615ecc4b23a3aa4395ae9a9156eb"}
Sep 08 11:44:27.551090 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:27.550922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb" event={"ID":"9e9fabd2-88a8-4522-8950-6a4d2248a5bd","Type":"ContainerStarted","Data":"063689a9f6deca8ec04097e06719d8ed9b01ce2f8b4b90acbc34408952f12141"}
Sep 08 11:44:27.551090 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:27.551022    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"
Sep 08 11:44:27.572266 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:27.572212    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb" podStartSLOduration=1.5721954 podStartE2EDuration="1.5721954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:44:27.570268718 +0000 UTC m=+874.137029272" watchObservedRunningTime="2026-09-08 11:44:27.5721954 +0000 UTC m=+874.138955998"
Sep 08 11:44:32.407117 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.407080    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"]
Sep 08 11:44:32.410499 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.410459    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.413047 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.413020    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-b2e6067b-kserve-self-signed-certs\""
Sep 08 11:44:32.422456 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.422434    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"]
Sep 08 11:44:32.499281 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.499246    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tmp-dir\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.499281 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.499283    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpdj\" (UniqueName: \"kubernetes.io/projected/7f59f5f7-6fa1-4b0d-87c4-794add332a99-kube-api-access-nqpdj\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.499493 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.499346    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-home\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.499493 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.499394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-dshm\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.499493 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.499453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-model-cache\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.499602 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.499500    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.600309 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.600274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-model-cache\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.600309 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.600310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.600573 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.600332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tmp-dir\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.600573 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.600353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpdj\" (UniqueName: \"kubernetes.io/projected/7f59f5f7-6fa1-4b0d-87c4-794add332a99-kube-api-access-nqpdj\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.600573 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.600403    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-home\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.600573 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.600439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-dshm\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.600776 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.600726    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-model-cache\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.600776 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.600758    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tmp-dir\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.600841 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.600822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-home\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.602570 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.602551    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-dshm\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.602890 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.602873    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.609987 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.609962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpdj\" (UniqueName: \"kubernetes.io/projected/7f59f5f7-6fa1-4b0d-87c4-794add332a99-kube-api-access-nqpdj\") pod \"e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") " pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.719578 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.719543    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:32.990815 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:32.990772    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"]
Sep 08 11:44:32.992943 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:44:32.992915    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f59f5f7_6fa1_4b0d_87c4_794add332a99.slice/crio-b5ff35fe579fef5fde5f933fc93e919cc2ce29962737ce0f3944d9a796e78462 WatchSource:0}: Error finding container b5ff35fe579fef5fde5f933fc93e919cc2ce29962737ce0f3944d9a796e78462: Status 404 returned error can't find the container with id b5ff35fe579fef5fde5f933fc93e919cc2ce29962737ce0f3944d9a796e78462
Sep 08 11:44:33.439166 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:33.439083    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:33.579238 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:33.579204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg" event={"ID":"7f59f5f7-6fa1-4b0d-87c4-794add332a99","Type":"ContainerStarted","Data":"920dfbc881d23a16d38da5fd57e1fb5a2e1a396fbb5730340ecc6655a0fd2dd5"}
Sep 08 11:44:33.579238 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:33.579244    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg" event={"ID":"7f59f5f7-6fa1-4b0d-87c4-794add332a99","Type":"ContainerStarted","Data":"b5ff35fe579fef5fde5f933fc93e919cc2ce29962737ce0f3944d9a796e78462"}
Sep 08 11:44:33.599572 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:33.599525    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg" podStartSLOduration=1.599507911 podStartE2EDuration="1.599507911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:44:33.59851909 +0000 UTC m=+880.165279697" watchObservedRunningTime="2026-09-08 11:44:33.599507911 +0000 UTC m=+880.166268453"
Sep 08 11:44:38.557442 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:38.557405    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"
Sep 08 11:44:41.021296 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.021258    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"]
Sep 08 11:44:41.024797 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.024775    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.028692 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.028529    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-95669b\"/\"kube-root-ca.crt\""
Sep 08 11:44:41.028692 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.028568    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-95669b\"/\"test-model-95669b-kserve-self-signed-certs\""
Sep 08 11:44:41.028692 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.028593    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-95669b\"/\"openshift-service-ca.crt\""
Sep 08 11:44:41.028692 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.028574    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-95669b\"/\"default-dockercfg-hj8cl\""
Sep 08 11:44:41.035437 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.035415    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"]
Sep 08 11:44:41.064622 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.064595    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:41.072967 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.072931    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-dshm\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.073116 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.072980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6bc8\" (UniqueName: \"kubernetes.io/projected/87e6aee8-8be7-4ca0-ac95-b001372a6b45-kube-api-access-r6bc8\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.073116 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.073077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-tmp-dir\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.073213 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.073135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e6aee8-8be7-4ca0-ac95-b001372a6b45-tls-certs\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.073213 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.073170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-model-cache\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.073291 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.073250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-home\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.174250 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.174209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-dshm\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.174250 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.174250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6bc8\" (UniqueName: \"kubernetes.io/projected/87e6aee8-8be7-4ca0-ac95-b001372a6b45-kube-api-access-r6bc8\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.174507 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.174305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-tmp-dir\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.174507 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.174333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e6aee8-8be7-4ca0-ac95-b001372a6b45-tls-certs\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.174507 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.174369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-model-cache\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.174507 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.174419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-home\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.174798 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.174774    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-tmp-dir\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.174873 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.174782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-home\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.174873 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.174854    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-model-cache\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.176476 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.176448    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-dshm\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.176882 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.176859    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e6aee8-8be7-4ca0-ac95-b001372a6b45-tls-certs\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.186912 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.186882    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6bc8\" (UniqueName: \"kubernetes.io/projected/87e6aee8-8be7-4ca0-ac95-b001372a6b45-kube-api-access-r6bc8\") pod \"test-model-95669b-kserve-5f69646f6c-qsjfs\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") " pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.360592 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.360503    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:41.634815 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:41.634789    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"]
Sep 08 11:44:41.639328 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:44:41.639296    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87e6aee8_8be7_4ca0_ac95_b001372a6b45.slice/crio-648163665d345e4fb6a1dd9085b712d7acc524f7012fed5a3a39f589416526dc WatchSource:0}: Error finding container 648163665d345e4fb6a1dd9085b712d7acc524f7012fed5a3a39f589416526dc: Status 404 returned error can't find the container with id 648163665d345e4fb6a1dd9085b712d7acc524f7012fed5a3a39f589416526dc
Sep 08 11:44:42.624578 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:42.624546    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs" event={"ID":"87e6aee8-8be7-4ca0-ac95-b001372a6b45","Type":"ContainerStarted","Data":"95b9d1d46abd24d5d7780c62777c047f011cb76786758d56b308bac2b79c0b18"}
Sep 08 11:44:42.624578 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:42.624584    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs" event={"ID":"87e6aee8-8be7-4ca0-ac95-b001372a6b45","Type":"ContainerStarted","Data":"648163665d345e4fb6a1dd9085b712d7acc524f7012fed5a3a39f589416526dc"}
Sep 08 11:44:42.646986 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:42.646927    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs" podStartSLOduration=1.646910944 podStartE2EDuration="1.646910944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:44:42.645448638 +0000 UTC m=+889.212209180" watchObservedRunningTime="2026-09-08 11:44:42.646910944 +0000 UTC m=+889.213671485"
Sep 08 11:44:42.720429 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:42.720393    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:42.720759 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:42.720729    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:42.733607 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:42.733575    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:43.218520 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.218490    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:43.227231 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.227200    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc"]
Sep 08 11:44:43.230694 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.230676    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc"
Sep 08 11:44:43.233175 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.233154    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:44:43.233283 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.233192    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:44:43.238762 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.238728    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc"]
Sep 08 11:44:43.295207 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.295165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99142382-45be-4409-9a8b-eb99fe410573-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a1721d-5j9mc\" (UID: \"99142382-45be-4409-9a8b-eb99fe410573\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc"
Sep 08 11:44:43.396428 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.396375    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99142382-45be-4409-9a8b-eb99fe410573-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a1721d-5j9mc\" (UID: \"99142382-45be-4409-9a8b-eb99fe410573\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc"
Sep 08 11:44:43.397068 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.397043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99142382-45be-4409-9a8b-eb99fe410573-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a1721d-5j9mc\" (UID: \"99142382-45be-4409-9a8b-eb99fe410573\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc"
Sep 08 11:44:43.473710 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.473620    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:43.544782 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.544742    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc"
Sep 08 11:44:43.640762 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.640683    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:44:43.700649 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:43.700616    2561 secret.go:189] Couldn't get secret llm/e2e-conflict-b-b2e6067b-kserve-self-signed-certs: secret "e2e-conflict-b-b2e6067b-kserve-self-signed-certs" not found
Sep 08 11:44:43.700809 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:43.700692    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs podName:7f59f5f7-6fa1-4b0d-87c4-794add332a99 nodeName:}" failed. No retries permitted until 2026-09-08 11:44:44.200670564 +0000 UTC m=+890.767431082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs") pod "e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg" (UID: "7f59f5f7-6fa1-4b0d-87c4-794add332a99") : secret "e2e-conflict-b-b2e6067b-kserve-self-signed-certs" not found
Sep 08 11:44:43.724923 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.724833    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"]
Sep 08 11:44:43.741076 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.741042    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:43.818509 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:43.818398    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc"]
Sep 08 11:44:43.820536 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:44:43.820511    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99142382_45be_4409_9a8b_eb99fe410573.slice/crio-37a24a3fd8c453ea214f664f23176b901c55b6e5d49ed376a3005d5b168d899c WatchSource:0}: Error finding container 37a24a3fd8c453ea214f664f23176b901c55b6e5d49ed376a3005d5b168d899c: Status 404 returned error can't find the container with id 37a24a3fd8c453ea214f664f23176b901c55b6e5d49ed376a3005d5b168d899c
Sep 08 11:44:44.205636 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:44.205591    2561 secret.go:189] Couldn't get secret llm/e2e-conflict-b-b2e6067b-kserve-self-signed-certs: secret "e2e-conflict-b-b2e6067b-kserve-self-signed-certs" not found
Sep 08 11:44:44.205842 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:44.205676    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs podName:7f59f5f7-6fa1-4b0d-87c4-794add332a99 nodeName:}" failed. No retries permitted until 2026-09-08 11:44:45.205657738 +0000 UTC m=+891.772418275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs") pod "e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg" (UID: "7f59f5f7-6fa1-4b0d-87c4-794add332a99") : secret "e2e-conflict-b-b2e6067b-kserve-self-signed-certs" not found
Sep 08 11:44:44.635245 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:44.634911    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc" event={"ID":"99142382-45be-4409-9a8b-eb99fe410573","Type":"ContainerStarted","Data":"0dda68215d53bcb36933c808f16aba9094dc48c43ae5d1142fe3c920a4ea5404"}
Sep 08 11:44:44.635245 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:44.634963    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc" event={"ID":"99142382-45be-4409-9a8b-eb99fe410573","Type":"ContainerStarted","Data":"37a24a3fd8c453ea214f664f23176b901c55b6e5d49ed376a3005d5b168d899c"}
Sep 08 11:44:44.654031 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:44.653911    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc" podStartSLOduration=1.653891147 podStartE2EDuration="1.653891147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:44:44.653419388 +0000 UTC m=+891.220179930" watchObservedRunningTime="2026-09-08 11:44:44.653891147 +0000 UTC m=+891.220651688"
Sep 08 11:44:45.215871 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:45.215832    2561 secret.go:189] Couldn't get secret llm/e2e-conflict-b-b2e6067b-kserve-self-signed-certs: secret "e2e-conflict-b-b2e6067b-kserve-self-signed-certs" not found
Sep 08 11:44:45.216050 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:45.215914    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs podName:7f59f5f7-6fa1-4b0d-87c4-794add332a99 nodeName:}" failed. No retries permitted until 2026-09-08 11:44:47.215898607 +0000 UTC m=+893.782659125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs") pod "e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg" (UID: "7f59f5f7-6fa1-4b0d-87c4-794add332a99") : secret "e2e-conflict-b-b2e6067b-kserve-self-signed-certs" not found
Sep 08 11:44:45.640989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:45.640897    2561 generic.go:356] "Generic (PLEG): container finished" podID="99142382-45be-4409-9a8b-eb99fe410573" containerID="0dda68215d53bcb36933c808f16aba9094dc48c43ae5d1142fe3c920a4ea5404" exitCode=0
Sep 08 11:44:45.640989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:45.640978    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc" event={"ID":"99142382-45be-4409-9a8b-eb99fe410573","Type":"ContainerDied","Data":"0dda68215d53bcb36933c808f16aba9094dc48c43ae5d1142fe3c920a4ea5404"}
Sep 08 11:44:46.552165 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:46.552045    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:46.768870 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:46.768840    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc"
Sep 08 11:44:46.832274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:46.832188    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99142382-45be-4409-9a8b-eb99fe410573-maas-api-service-ca\") pod \"99142382-45be-4409-9a8b-eb99fe410573\" (UID: \"99142382-45be-4409-9a8b-eb99fe410573\") "
Sep 08 11:44:46.832580 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:46.832554    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99142382-45be-4409-9a8b-eb99fe410573-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99142382-45be-4409-9a8b-eb99fe410573" (UID: "99142382-45be-4409-9a8b-eb99fe410573"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:44:46.933662 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:46.933626    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99142382-45be-4409-9a8b-eb99fe410573-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:44:47.237500 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:47.237447    2561 secret.go:189] Couldn't get secret llm/e2e-conflict-b-b2e6067b-kserve-self-signed-certs: secret "e2e-conflict-b-b2e6067b-kserve-self-signed-certs" not found
Sep 08 11:44:47.237714 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:47.237548    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs podName:7f59f5f7-6fa1-4b0d-87c4-794add332a99 nodeName:}" failed. No retries permitted until 2026-09-08 11:44:51.23752827 +0000 UTC m=+897.804288807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs") pod "e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg" (UID: "7f59f5f7-6fa1-4b0d-87c4-794add332a99") : secret "e2e-conflict-b-b2e6067b-kserve-self-signed-certs" not found
Sep 08 11:44:47.652654 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:47.652573    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc"
Sep 08 11:44:47.652654 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:47.652586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc" event={"ID":"99142382-45be-4409-9a8b-eb99fe410573","Type":"ContainerDied","Data":"37a24a3fd8c453ea214f664f23176b901c55b6e5d49ed376a3005d5b168d899c"}
Sep 08 11:44:47.652654 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:47.652636    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37a24a3fd8c453ea214f664f23176b901c55b6e5d49ed376a3005d5b168d899c"
Sep 08 11:44:50.046424 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.046389    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7"]
Sep 08 11:44:50.046941 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.046923    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99142382-45be-4409-9a8b-eb99fe410573" containerName="revoke-keys"
Sep 08 11:44:50.046991 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.046944    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="99142382-45be-4409-9a8b-eb99fe410573" containerName="revoke-keys"
Sep 08 11:44:50.047057 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.047046    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="99142382-45be-4409-9a8b-eb99fe410573" containerName="revoke-keys"
Sep 08 11:44:50.056973 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.056944    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7"
Sep 08 11:44:50.057323 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.057295    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7"]
Sep 08 11:44:50.059868 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.059845    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:44:50.060007 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.059849    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:44:50.166230 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.166192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59de0b45-ea7f-4822-b484-4da37a8f2871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7\" (UID: \"59de0b45-ea7f-4822-b484-4da37a8f2871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7"
Sep 08 11:44:50.267196 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.267161    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59de0b45-ea7f-4822-b484-4da37a8f2871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7\" (UID: \"59de0b45-ea7f-4822-b484-4da37a8f2871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7"
Sep 08 11:44:50.267808 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.267789    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59de0b45-ea7f-4822-b484-4da37a8f2871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7\" (UID: \"59de0b45-ea7f-4822-b484-4da37a8f2871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7"
Sep 08 11:44:50.368148 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.368050    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7"
Sep 08 11:44:50.639622 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.639534    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7"]
Sep 08 11:44:50.643535 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:44:50.643511    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59de0b45_ea7f_4822_b484_4da37a8f2871.slice/crio-9135ee57d3d0378d19a06297408e21e8650c5297e8355f4c4bfc382221bd0232 WatchSource:0}: Error finding container 9135ee57d3d0378d19a06297408e21e8650c5297e8355f4c4bfc382221bd0232: Status 404 returned error can't find the container with id 9135ee57d3d0378d19a06297408e21e8650c5297e8355f4c4bfc382221bd0232
Sep 08 11:44:50.670524 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:50.670492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7" event={"ID":"59de0b45-ea7f-4822-b484-4da37a8f2871","Type":"ContainerStarted","Data":"9135ee57d3d0378d19a06297408e21e8650c5297e8355f4c4bfc382221bd0232"}
Sep 08 11:44:51.276669 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:51.276630    2561 secret.go:189] Couldn't get secret llm/e2e-conflict-b-b2e6067b-kserve-self-signed-certs: secret "e2e-conflict-b-b2e6067b-kserve-self-signed-certs" not found
Sep 08 11:44:51.277055 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:51.276703    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs podName:7f59f5f7-6fa1-4b0d-87c4-794add332a99 nodeName:}" failed. No retries permitted until 2026-09-08 11:44:59.276686333 +0000 UTC m=+905.843446870 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs") pod "e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg" (UID: "7f59f5f7-6fa1-4b0d-87c4-794add332a99") : secret "e2e-conflict-b-b2e6067b-kserve-self-signed-certs" not found
Sep 08 11:44:51.361002 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:51.360969    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:51.361153 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:51.361018    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:51.373972 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:51.373942    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:51.676049 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:51.675958    2561 generic.go:356] "Generic (PLEG): container finished" podID="59de0b45-ea7f-4822-b484-4da37a8f2871" containerID="042bda6d2466c80a1958bebee18bdc89d8d5e4edcfd61c6f1c59841dfcd2e6dd" exitCode=0
Sep 08 11:44:51.676220 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:51.676048    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7" event={"ID":"59de0b45-ea7f-4822-b484-4da37a8f2871","Type":"ContainerDied","Data":"042bda6d2466c80a1958bebee18bdc89d8d5e4edcfd61c6f1c59841dfcd2e6dd"}
Sep 08 11:44:51.687857 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:51.687835    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:44:51.881866 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:51.881832    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:52.812869 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:52.812847    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7"
Sep 08 11:44:52.890969 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:52.890936    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59de0b45-ea7f-4822-b484-4da37a8f2871-maas-api-service-ca\") pod \"59de0b45-ea7f-4822-b484-4da37a8f2871\" (UID: \"59de0b45-ea7f-4822-b484-4da37a8f2871\") "
Sep 08 11:44:52.891337 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:52.891317    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59de0b45-ea7f-4822-b484-4da37a8f2871-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59de0b45-ea7f-4822-b484-4da37a8f2871" (UID: "59de0b45-ea7f-4822-b484-4da37a8f2871"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:44:52.992529 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:52.992491    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59de0b45-ea7f-4822-b484-4da37a8f2871-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:44:53.392371 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:53.392283    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:53.688916 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:53.688836    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7"
Sep 08 11:44:53.688916 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:53.688859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7" event={"ID":"59de0b45-ea7f-4822-b484-4da37a8f2871","Type":"ContainerDied","Data":"9135ee57d3d0378d19a06297408e21e8650c5297e8355f4c4bfc382221bd0232"}
Sep 08 11:44:53.688916 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:53.688904    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9135ee57d3d0378d19a06297408e21e8650c5297e8355f4c4bfc382221bd0232"
Sep 08 11:44:54.046215 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.046174    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:54.079251 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.079217    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"]
Sep 08 11:44:54.079504 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.079433    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89" podUID="fec32d74-55da-43c6-804d-ef8976022e60" containerName="payload-processing" containerID="cri-o://eb267fd1de6b240fa3ee2ef6ef8bab9969dcc5c4653f82f854fd9aa7ea0564f6" gracePeriod=30
Sep 08 11:44:54.088153 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.087977    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"]
Sep 08 11:44:54.088317 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.088246    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr" podUID="427ac17c-5f57-4a92-b88e-02428119fb80" containerName="payload-pre-processing" containerID="cri-o://4593188b2a415cc93c31d478e66383b7b5bba07dfadab1f4800e1516b4b9fd92" gracePeriod=30
Sep 08 11:44:54.130967 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.130925    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:54.177549 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.177506    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89" podUID="fec32d74-55da-43c6-804d-ef8976022e60" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.40:9004: connect: connection refused"
Sep 08 11:44:54.177673 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.177503    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr" podUID="427ac17c-5f57-4a92-b88e-02428119fb80" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.39:9004: connect: connection refused"
Sep 08 11:44:54.213999 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.213968    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:54.288379 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.288240    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc"]
Sep 08 11:44:54.290576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.290436    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:54.291005 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.290977    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a1721d-5j9mc"]
Sep 08 11:44:54.359722 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.359698    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:54.362008 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.361988    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"
Sep 08 11:44:54.364787 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.364770    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"
Sep 08 11:44:54.507418 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.507383    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6js4h\" (UniqueName: \"kubernetes.io/projected/fec32d74-55da-43c6-804d-ef8976022e60-kube-api-access-6js4h\") pod \"fec32d74-55da-43c6-804d-ef8976022e60\" (UID: \"fec32d74-55da-43c6-804d-ef8976022e60\") "
Sep 08 11:44:54.507611 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.507459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427ac17c-5f57-4a92-b88e-02428119fb80-plugins-config-volume\") pod \"427ac17c-5f57-4a92-b88e-02428119fb80\" (UID: \"427ac17c-5f57-4a92-b88e-02428119fb80\") "
Sep 08 11:44:54.507611 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.507523    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec32d74-55da-43c6-804d-ef8976022e60-plugins-config-volume\") pod \"fec32d74-55da-43c6-804d-ef8976022e60\" (UID: \"fec32d74-55da-43c6-804d-ef8976022e60\") "
Sep 08 11:44:54.507611 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.507554    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rghjc\" (UniqueName: \"kubernetes.io/projected/427ac17c-5f57-4a92-b88e-02428119fb80-kube-api-access-rghjc\") pod \"427ac17c-5f57-4a92-b88e-02428119fb80\" (UID: \"427ac17c-5f57-4a92-b88e-02428119fb80\") "
Sep 08 11:44:54.507851 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.507827    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec32d74-55da-43c6-804d-ef8976022e60-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fec32d74-55da-43c6-804d-ef8976022e60" (UID: "fec32d74-55da-43c6-804d-ef8976022e60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:44:54.507910 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.507842    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427ac17c-5f57-4a92-b88e-02428119fb80-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "427ac17c-5f57-4a92-b88e-02428119fb80" (UID: "427ac17c-5f57-4a92-b88e-02428119fb80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:44:54.509566 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.509544    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec32d74-55da-43c6-804d-ef8976022e60-kube-api-access-6js4h" (OuterVolumeSpecName: "kube-api-access-6js4h") pod "fec32d74-55da-43c6-804d-ef8976022e60" (UID: "fec32d74-55da-43c6-804d-ef8976022e60"). InnerVolumeSpecName "kube-api-access-6js4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:44:54.509662 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.509645    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427ac17c-5f57-4a92-b88e-02428119fb80-kube-api-access-rghjc" (OuterVolumeSpecName: "kube-api-access-rghjc") pod "427ac17c-5f57-4a92-b88e-02428119fb80" (UID: "427ac17c-5f57-4a92-b88e-02428119fb80"). InnerVolumeSpecName "kube-api-access-rghjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:44:54.608363 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.608280    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fec32d74-55da-43c6-804d-ef8976022e60-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:44:54.608363 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.608312    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rghjc\" (UniqueName: \"kubernetes.io/projected/427ac17c-5f57-4a92-b88e-02428119fb80-kube-api-access-rghjc\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:44:54.608363 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.608323    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6js4h\" (UniqueName: \"kubernetes.io/projected/fec32d74-55da-43c6-804d-ef8976022e60-kube-api-access-6js4h\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:44:54.608363 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.608332    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427ac17c-5f57-4a92-b88e-02428119fb80-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:44:54.636159 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.636123    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:54.695260 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.695225    2561 generic.go:356] "Generic (PLEG): container finished" podID="fec32d74-55da-43c6-804d-ef8976022e60" containerID="eb267fd1de6b240fa3ee2ef6ef8bab9969dcc5c4653f82f854fd9aa7ea0564f6" exitCode=0
Sep 08 11:44:54.695397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.695291    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"
Sep 08 11:44:54.695397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.695309    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89" event={"ID":"fec32d74-55da-43c6-804d-ef8976022e60","Type":"ContainerDied","Data":"eb267fd1de6b240fa3ee2ef6ef8bab9969dcc5c4653f82f854fd9aa7ea0564f6"}
Sep 08 11:44:54.695397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.695361    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89" event={"ID":"fec32d74-55da-43c6-804d-ef8976022e60","Type":"ContainerDied","Data":"fb93b712d5a5aad008ff0b66fd0a0855f2b599d92b4f0f55616fc4014ce21f94"}
Sep 08 11:44:54.695397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.695387    2561 scope.go:117] "RemoveContainer" containerID="eb267fd1de6b240fa3ee2ef6ef8bab9969dcc5c4653f82f854fd9aa7ea0564f6"
Sep 08 11:44:54.696713 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.696693    2561 generic.go:356] "Generic (PLEG): container finished" podID="427ac17c-5f57-4a92-b88e-02428119fb80" containerID="4593188b2a415cc93c31d478e66383b7b5bba07dfadab1f4800e1516b4b9fd92" exitCode=0
Sep 08 11:44:54.696823 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.696747    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"
Sep 08 11:44:54.696823 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.696769    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr" event={"ID":"427ac17c-5f57-4a92-b88e-02428119fb80","Type":"ContainerDied","Data":"4593188b2a415cc93c31d478e66383b7b5bba07dfadab1f4800e1516b4b9fd92"}
Sep 08 11:44:54.696823 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.696817    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr" event={"ID":"427ac17c-5f57-4a92-b88e-02428119fb80","Type":"ContainerDied","Data":"80678c4e72f00c810bbc1f4005c6c31b650324ff677fe130e9f870c4587f7eea"}
Sep 08 11:44:54.703797 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.703779    2561 scope.go:117] "RemoveContainer" containerID="eb267fd1de6b240fa3ee2ef6ef8bab9969dcc5c4653f82f854fd9aa7ea0564f6"
Sep 08 11:44:54.704056 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:54.704036    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb267fd1de6b240fa3ee2ef6ef8bab9969dcc5c4653f82f854fd9aa7ea0564f6\": container with ID starting with eb267fd1de6b240fa3ee2ef6ef8bab9969dcc5c4653f82f854fd9aa7ea0564f6 not found: ID does not exist" containerID="eb267fd1de6b240fa3ee2ef6ef8bab9969dcc5c4653f82f854fd9aa7ea0564f6"
Sep 08 11:44:54.704141 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.704065    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb267fd1de6b240fa3ee2ef6ef8bab9969dcc5c4653f82f854fd9aa7ea0564f6"} err="failed to get container status \"eb267fd1de6b240fa3ee2ef6ef8bab9969dcc5c4653f82f854fd9aa7ea0564f6\": rpc error: code = NotFound desc = could not find container \"eb267fd1de6b240fa3ee2ef6ef8bab9969dcc5c4653f82f854fd9aa7ea0564f6\": container with ID starting with eb267fd1de6b240fa3ee2ef6ef8bab9969dcc5c4653f82f854fd9aa7ea0564f6 not found: ID does not exist"
Sep 08 11:44:54.704141 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.704080    2561 scope.go:117] "RemoveContainer" containerID="4593188b2a415cc93c31d478e66383b7b5bba07dfadab1f4800e1516b4b9fd92"
Sep 08 11:44:54.712053 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.712038    2561 scope.go:117] "RemoveContainer" containerID="4593188b2a415cc93c31d478e66383b7b5bba07dfadab1f4800e1516b4b9fd92"
Sep 08 11:44:54.712331 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:54.712311    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4593188b2a415cc93c31d478e66383b7b5bba07dfadab1f4800e1516b4b9fd92\": container with ID starting with 4593188b2a415cc93c31d478e66383b7b5bba07dfadab1f4800e1516b4b9fd92 not found: ID does not exist" containerID="4593188b2a415cc93c31d478e66383b7b5bba07dfadab1f4800e1516b4b9fd92"
Sep 08 11:44:54.712415 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.712336    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4593188b2a415cc93c31d478e66383b7b5bba07dfadab1f4800e1516b4b9fd92"} err="failed to get container status \"4593188b2a415cc93c31d478e66383b7b5bba07dfadab1f4800e1516b4b9fd92\": rpc error: code = NotFound desc = could not find container \"4593188b2a415cc93c31d478e66383b7b5bba07dfadab1f4800e1516b4b9fd92\": container with ID starting with 4593188b2a415cc93c31d478e66383b7b5bba07dfadab1f4800e1516b4b9fd92 not found: ID does not exist"
Sep 08 11:44:54.722819 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.722784    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"]
Sep 08 11:44:54.724638 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.724619    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a1721d-5b494b4fb5-plf89"]
Sep 08 11:44:54.738082 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.738059    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"]
Sep 08 11:44:54.742789 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:54.742769    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a1721d-6457dcc9f4-c4dgr"]
Sep 08 11:44:55.748002 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:55.747968    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:55.960826 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:55.960791    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="427ac17c-5f57-4a92-b88e-02428119fb80" path="/var/lib/kubelet/pods/427ac17c-5f57-4a92-b88e-02428119fb80/volumes"
Sep 08 11:44:55.961153 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:55.961139    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99142382-45be-4409-9a8b-eb99fe410573" path="/var/lib/kubelet/pods/99142382-45be-4409-9a8b-eb99fe410573/volumes"
Sep 08 11:44:55.961449 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:55.961437    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec32d74-55da-43c6-804d-ef8976022e60" path="/var/lib/kubelet/pods/fec32d74-55da-43c6-804d-ef8976022e60/volumes"
Sep 08 11:44:56.578177 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:56.578140    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:56.612862 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:56.612831    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"]
Sep 08 11:44:56.613111 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:56.613086    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb" podUID="9e9fabd2-88a8-4522-8950-6a4d2248a5bd" containerName="payload-processing" containerID="cri-o://1133e44f548f2a36c286c88987aabb7ff5ef615ecc4b23a3aa4395ae9a9156eb" gracePeriod=30
Sep 08 11:44:56.683031 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:56.682993    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:56.770915 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:56.770880    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7"]
Sep 08 11:44:56.774419 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:56.774395    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b16466d7-gsmj7"]
Sep 08 11:44:56.774740 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:56.774685    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:56.849454 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:56.849426    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:56.868062 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:56.868038    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"
Sep 08 11:44:57.032052 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.032015    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbgvp\" (UniqueName: \"kubernetes.io/projected/9e9fabd2-88a8-4522-8950-6a4d2248a5bd-kube-api-access-mbgvp\") pod \"9e9fabd2-88a8-4522-8950-6a4d2248a5bd\" (UID: \"9e9fabd2-88a8-4522-8950-6a4d2248a5bd\") "
Sep 08 11:44:57.032231 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.032086    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9fabd2-88a8-4522-8950-6a4d2248a5bd-plugins-config-volume\") pod \"9e9fabd2-88a8-4522-8950-6a4d2248a5bd\" (UID: \"9e9fabd2-88a8-4522-8950-6a4d2248a5bd\") "
Sep 08 11:44:57.032460 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.032436    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e9fabd2-88a8-4522-8950-6a4d2248a5bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e9fabd2-88a8-4522-8950-6a4d2248a5bd" (UID: "9e9fabd2-88a8-4522-8950-6a4d2248a5bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:44:57.034128 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.034103    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9fabd2-88a8-4522-8950-6a4d2248a5bd-kube-api-access-mbgvp" (OuterVolumeSpecName: "kube-api-access-mbgvp") pod "9e9fabd2-88a8-4522-8950-6a4d2248a5bd" (UID: "9e9fabd2-88a8-4522-8950-6a4d2248a5bd"). InnerVolumeSpecName "kube-api-access-mbgvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:44:57.132919 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.132839    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbgvp\" (UniqueName: \"kubernetes.io/projected/9e9fabd2-88a8-4522-8950-6a4d2248a5bd-kube-api-access-mbgvp\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:44:57.132919 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.132865    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9fabd2-88a8-4522-8950-6a4d2248a5bd-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:44:57.714860 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.714827    2561 generic.go:356] "Generic (PLEG): container finished" podID="9e9fabd2-88a8-4522-8950-6a4d2248a5bd" containerID="1133e44f548f2a36c286c88987aabb7ff5ef615ecc4b23a3aa4395ae9a9156eb" exitCode=0
Sep 08 11:44:57.715028 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.714867    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"
Sep 08 11:44:57.715028 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.714869    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb" event={"ID":"9e9fabd2-88a8-4522-8950-6a4d2248a5bd","Type":"ContainerDied","Data":"1133e44f548f2a36c286c88987aabb7ff5ef615ecc4b23a3aa4395ae9a9156eb"}
Sep 08 11:44:57.715028 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.714997    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb" event={"ID":"9e9fabd2-88a8-4522-8950-6a4d2248a5bd","Type":"ContainerDied","Data":"063689a9f6deca8ec04097e06719d8ed9b01ce2f8b4b90acbc34408952f12141"}
Sep 08 11:44:57.715129 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.715029    2561 scope.go:117] "RemoveContainer" containerID="1133e44f548f2a36c286c88987aabb7ff5ef615ecc4b23a3aa4395ae9a9156eb"
Sep 08 11:44:57.723575 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.723558    2561 scope.go:117] "RemoveContainer" containerID="1133e44f548f2a36c286c88987aabb7ff5ef615ecc4b23a3aa4395ae9a9156eb"
Sep 08 11:44:57.723857 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:57.723841    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1133e44f548f2a36c286c88987aabb7ff5ef615ecc4b23a3aa4395ae9a9156eb\": container with ID starting with 1133e44f548f2a36c286c88987aabb7ff5ef615ecc4b23a3aa4395ae9a9156eb not found: ID does not exist" containerID="1133e44f548f2a36c286c88987aabb7ff5ef615ecc4b23a3aa4395ae9a9156eb"
Sep 08 11:44:57.723906 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.723864    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1133e44f548f2a36c286c88987aabb7ff5ef615ecc4b23a3aa4395ae9a9156eb"} err="failed to get container status \"1133e44f548f2a36c286c88987aabb7ff5ef615ecc4b23a3aa4395ae9a9156eb\": rpc error: code = NotFound desc = could not find container \"1133e44f548f2a36c286c88987aabb7ff5ef615ecc4b23a3aa4395ae9a9156eb\": container with ID starting with 1133e44f548f2a36c286c88987aabb7ff5ef615ecc4b23a3aa4395ae9a9156eb not found: ID does not exist"
Sep 08 11:44:57.736513 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.736488    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"]
Sep 08 11:44:57.743801 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.743778    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b16466d7-6896757b95-fxbhb"]
Sep 08 11:44:57.914815 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.914782    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:57.960861 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.960830    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59de0b45-ea7f-4822-b484-4da37a8f2871" path="/var/lib/kubelet/pods/59de0b45-ea7f-4822-b484-4da37a8f2871/volumes"
Sep 08 11:44:57.961174 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:57.961161    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e9fabd2-88a8-4522-8950-6a4d2248a5bd" path="/var/lib/kubelet/pods/9e9fabd2-88a8-4522-8950-6a4d2248a5bd/volumes"
Sep 08 11:44:58.324344 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:58.324309    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:58.659453 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:58.659369    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:58.754230 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:44:58.754198    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:44:59.353479 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:59.353442    2561 secret.go:189] Couldn't get secret llm/e2e-conflict-b-b2e6067b-kserve-self-signed-certs: secret "e2e-conflict-b-b2e6067b-kserve-self-signed-certs" not found
Sep 08 11:44:59.353870 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:44:59.353546    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs podName:7f59f5f7-6fa1-4b0d-87c4-794add332a99 nodeName:}" failed. No retries permitted until 2026-09-08 11:45:15.353531372 +0000 UTC m=+921.920291890 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs") pod "e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg" (UID: "7f59f5f7-6fa1-4b0d-87c4-794add332a99") : secret "e2e-conflict-b-b2e6067b-kserve-self-signed-certs" not found
Sep 08 11:45:00.145518 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.145484    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-425b2c-29814465-k29ht"]
Sep 08 11:45:00.145836 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.145823    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59de0b45-ea7f-4822-b484-4da37a8f2871" containerName="revoke-keys"
Sep 08 11:45:00.145836 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.145837    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="59de0b45-ea7f-4822-b484-4da37a8f2871" containerName="revoke-keys"
Sep 08 11:45:00.145914 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.145852    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e9fabd2-88a8-4522-8950-6a4d2248a5bd" containerName="payload-processing"
Sep 08 11:45:00.145914 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.145858    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e9fabd2-88a8-4522-8950-6a4d2248a5bd" containerName="payload-processing"
Sep 08 11:45:00.145914 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.145867    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec32d74-55da-43c6-804d-ef8976022e60" containerName="payload-processing"
Sep 08 11:45:00.145914 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.145873    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec32d74-55da-43c6-804d-ef8976022e60" containerName="payload-processing"
Sep 08 11:45:00.145914 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.145881    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="427ac17c-5f57-4a92-b88e-02428119fb80" containerName="payload-pre-processing"
Sep 08 11:45:00.145914 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.145886    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="427ac17c-5f57-4a92-b88e-02428119fb80" containerName="payload-pre-processing"
Sep 08 11:45:00.146162 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.145940    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e9fabd2-88a8-4522-8950-6a4d2248a5bd" containerName="payload-processing"
Sep 08 11:45:00.146162 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.145950    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="427ac17c-5f57-4a92-b88e-02428119fb80" containerName="payload-pre-processing"
Sep 08 11:45:00.146162 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.145956    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="59de0b45-ea7f-4822-b484-4da37a8f2871" containerName="revoke-keys"
Sep 08 11:45:00.146162 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.145961    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec32d74-55da-43c6-804d-ef8976022e60" containerName="payload-processing"
Sep 08 11:45:00.150203 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.150180    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814465-g8hnd"]
Sep 08 11:45:00.150322 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.150288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-425b2c-29814465-k29ht"
Sep 08 11:45:00.153497 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.153477    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814465-g8hnd"
Sep 08 11:45:00.153647 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.153629    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:45:00.158566 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.158545    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-425b2c-29814465-k29ht"]
Sep 08 11:45:00.163044 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.163025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-425b2c-29814465-k29ht"
Sep 08 11:45:00.163415 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.163392    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814465-g8hnd"]
Sep 08 11:45:00.171582 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.171563    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814465-g8hnd"
Sep 08 11:45:00.455798 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.455770    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-425b2c-29814465-k29ht"]
Sep 08 11:45:00.457323 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.457288    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814465-g8hnd"]
Sep 08 11:45:00.457810 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:45:00.457789    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75ee979_910d_4bb8_bf75_f3dba2416c46.slice/crio-c21932a1f816d61cf389950305d532b86c0f74e0ad339b819f4a8075626de17f WatchSource:0}: Error finding container c21932a1f816d61cf389950305d532b86c0f74e0ad339b819f4a8075626de17f: Status 404 returned error can't find the container with id c21932a1f816d61cf389950305d532b86c0f74e0ad339b819f4a8075626de17f
Sep 08 11:45:00.458784 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:45:00.458764    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6188b923_04db_43c4_b39f_970012e22779.slice/crio-acb74755d481b261321c9bdae1a4d5b1d44f46ba357634fca0d9584a424c3725 WatchSource:0}: Error finding container acb74755d481b261321c9bdae1a4d5b1d44f46ba357634fca0d9584a424c3725: Status 404 returned error can't find the container with id acb74755d481b261321c9bdae1a4d5b1d44f46ba357634fca0d9584a424c3725
Sep 08 11:45:00.662663 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.662627    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg" podUID="7f59f5f7-6fa1-4b0d-87c4-794add332a99" containerName="main" containerID="cri-o://920dfbc881d23a16d38da5fd57e1fb5a2e1a396fbb5730340ecc6655a0fd2dd5" gracePeriod=45
Sep 08 11:45:00.731712 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.731605    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-425b2c-29814465-k29ht" event={"ID":"a75ee979-910d-4bb8-bf75-f3dba2416c46","Type":"ContainerStarted","Data":"c21932a1f816d61cf389950305d532b86c0f74e0ad339b819f4a8075626de17f"}
Sep 08 11:45:00.732806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.732780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814465-g8hnd" event={"ID":"6188b923-04db-43c4-b39f-970012e22779","Type":"ContainerStarted","Data":"acb74755d481b261321c9bdae1a4d5b1d44f46ba357634fca0d9584a424c3725"}
Sep 08 11:45:00.891988 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.891963    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:45:00.966309 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.966271    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-home\") pod \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") "
Sep 08 11:45:00.966502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.966345    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-dshm\") pod \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") "
Sep 08 11:45:00.966502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.966404    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs\") pod \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") "
Sep 08 11:45:00.966502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.966454    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqpdj\" (UniqueName: \"kubernetes.io/projected/7f59f5f7-6fa1-4b0d-87c4-794add332a99-kube-api-access-nqpdj\") pod \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") "
Sep 08 11:45:00.966502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.966500    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-model-cache\") pod \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") "
Sep 08 11:45:00.966734 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.966538    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tmp-dir\") pod \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\" (UID: \"7f59f5f7-6fa1-4b0d-87c4-794add332a99\") "
Sep 08 11:45:00.966734 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.966557    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-home" (OuterVolumeSpecName: "home") pod "7f59f5f7-6fa1-4b0d-87c4-794add332a99" (UID: "7f59f5f7-6fa1-4b0d-87c4-794add332a99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:45:00.966838 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.966750    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f59f5f7-6fa1-4b0d-87c4-794add332a99" (UID: "7f59f5f7-6fa1-4b0d-87c4-794add332a99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:45:00.966838 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.966759    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-home\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:45:00.966922 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.966842    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f59f5f7-6fa1-4b0d-87c4-794add332a99" (UID: "7f59f5f7-6fa1-4b0d-87c4-794add332a99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:45:00.968541 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.968520    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-dshm" (OuterVolumeSpecName: "dshm") pod "7f59f5f7-6fa1-4b0d-87c4-794add332a99" (UID: "7f59f5f7-6fa1-4b0d-87c4-794add332a99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:45:00.968621 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.968579    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f59f5f7-6fa1-4b0d-87c4-794add332a99" (UID: "7f59f5f7-6fa1-4b0d-87c4-794add332a99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:45:00.968621 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:00.968601    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f59f5f7-6fa1-4b0d-87c4-794add332a99-kube-api-access-nqpdj" (OuterVolumeSpecName: "kube-api-access-nqpdj") pod "7f59f5f7-6fa1-4b0d-87c4-794add332a99" (UID: "7f59f5f7-6fa1-4b0d-87c4-794add332a99"). InnerVolumeSpecName "kube-api-access-nqpdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:45:01.067829 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.067743    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqpdj\" (UniqueName: \"kubernetes.io/projected/7f59f5f7-6fa1-4b0d-87c4-794add332a99-kube-api-access-nqpdj\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:45:01.067829 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.067770    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-model-cache\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:45:01.067829 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.067779    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tmp-dir\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:45:01.067829 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.067788    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f59f5f7-6fa1-4b0d-87c4-794add332a99-dshm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:45:01.067829 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.067796    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f59f5f7-6fa1-4b0d-87c4-794add332a99-tls-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:45:01.540543 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.540506    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:01.646119 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.646085    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:01.743916 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.743882    2561 generic.go:356] "Generic (PLEG): container finished" podID="a75ee979-910d-4bb8-bf75-f3dba2416c46" containerID="0055c7dd031376fc5468becceef8e9fb00f8f536ccb0a66d0dfb1b8dc21c5ee9" exitCode=0
Sep 08 11:45:01.744103 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.743965    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-425b2c-29814465-k29ht" event={"ID":"a75ee979-910d-4bb8-bf75-f3dba2416c46","Type":"ContainerDied","Data":"0055c7dd031376fc5468becceef8e9fb00f8f536ccb0a66d0dfb1b8dc21c5ee9"}
Sep 08 11:45:01.745197 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.745175    2561 generic.go:356] "Generic (PLEG): container finished" podID="7f59f5f7-6fa1-4b0d-87c4-794add332a99" containerID="920dfbc881d23a16d38da5fd57e1fb5a2e1a396fbb5730340ecc6655a0fd2dd5" exitCode=0
Sep 08 11:45:01.745284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.745217    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"
Sep 08 11:45:01.745284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.745234    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg" event={"ID":"7f59f5f7-6fa1-4b0d-87c4-794add332a99","Type":"ContainerDied","Data":"920dfbc881d23a16d38da5fd57e1fb5a2e1a396fbb5730340ecc6655a0fd2dd5"}
Sep 08 11:45:01.745284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.745262    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg" event={"ID":"7f59f5f7-6fa1-4b0d-87c4-794add332a99","Type":"ContainerDied","Data":"b5ff35fe579fef5fde5f933fc93e919cc2ce29962737ce0f3944d9a796e78462"}
Sep 08 11:45:01.745284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.745281    2561 scope.go:117] "RemoveContainer" containerID="920dfbc881d23a16d38da5fd57e1fb5a2e1a396fbb5730340ecc6655a0fd2dd5"
Sep 08 11:45:01.747053 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.747027    2561 generic.go:356] "Generic (PLEG): container finished" podID="6188b923-04db-43c4-b39f-970012e22779" containerID="a82bbc161372c5b3cc328641a5eebebe7339b4ff2e9a33985eae3e158a2da3c6" exitCode=0
Sep 08 11:45:01.747138 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.747057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814465-g8hnd" event={"ID":"6188b923-04db-43c4-b39f-970012e22779","Type":"ContainerDied","Data":"a82bbc161372c5b3cc328641a5eebebe7339b4ff2e9a33985eae3e158a2da3c6"}
Sep 08 11:45:01.754611 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.754595    2561 scope.go:117] "RemoveContainer" containerID="920dfbc881d23a16d38da5fd57e1fb5a2e1a396fbb5730340ecc6655a0fd2dd5"
Sep 08 11:45:01.754858 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:45:01.754841    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920dfbc881d23a16d38da5fd57e1fb5a2e1a396fbb5730340ecc6655a0fd2dd5\": container with ID starting with 920dfbc881d23a16d38da5fd57e1fb5a2e1a396fbb5730340ecc6655a0fd2dd5 not found: ID does not exist" containerID="920dfbc881d23a16d38da5fd57e1fb5a2e1a396fbb5730340ecc6655a0fd2dd5"
Sep 08 11:45:01.754898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.754867    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920dfbc881d23a16d38da5fd57e1fb5a2e1a396fbb5730340ecc6655a0fd2dd5"} err="failed to get container status \"920dfbc881d23a16d38da5fd57e1fb5a2e1a396fbb5730340ecc6655a0fd2dd5\": rpc error: code = NotFound desc = could not find container \"920dfbc881d23a16d38da5fd57e1fb5a2e1a396fbb5730340ecc6655a0fd2dd5\": container with ID starting with 920dfbc881d23a16d38da5fd57e1fb5a2e1a396fbb5730340ecc6655a0fd2dd5 not found: ID does not exist"
Sep 08 11:45:01.783785 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.783757    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"]
Sep 08 11:45:01.786576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.786517    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-b2e6067b-kserve-6ff88c748c-cvvvg"]
Sep 08 11:45:01.822173 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.822140    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"]
Sep 08 11:45:01.822488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.822462    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f59f5f7-6fa1-4b0d-87c4-794add332a99" containerName="main"
Sep 08 11:45:01.822538 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.822490    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f59f5f7-6fa1-4b0d-87c4-794add332a99" containerName="main"
Sep 08 11:45:01.822575 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.822552    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f59f5f7-6fa1-4b0d-87c4-794add332a99" containerName="main"
Sep 08 11:45:01.826814 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.826795    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.832671 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.832646    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-425b2c\"/\"kube-root-ca.crt\""
Sep 08 11:45:01.832774 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.832708    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-425b2c\"/\"test-model-425b2c-kserve-self-signed-certs\""
Sep 08 11:45:01.832969 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.832953    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-425b2c\"/\"openshift-service-ca.crt\""
Sep 08 11:45:01.833041 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.833013    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-425b2c\"/\"default-dockercfg-jm9sh\""
Sep 08 11:45:01.839000 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.838978    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"]
Sep 08 11:45:01.860859 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.860833    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:01.874281 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.874257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-model-cache\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.874387 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.874289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-tls-certs\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.874387 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.874345    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-dshm\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.874459 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.874402    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-home\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.874459 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.874433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22snp\" (UniqueName: \"kubernetes.io/projected/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-kube-api-access-22snp\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.874600 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.874549    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-tmp-dir\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.962062 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.962026    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f59f5f7-6fa1-4b0d-87c4-794add332a99" path="/var/lib/kubelet/pods/7f59f5f7-6fa1-4b0d-87c4-794add332a99/volumes"
Sep 08 11:45:01.975234 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.975201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-home\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.975361 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.975250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22snp\" (UniqueName: \"kubernetes.io/projected/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-kube-api-access-22snp\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.975361 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.975301    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-tmp-dir\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.975361 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.975350    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-model-cache\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.975547 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.975374    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-tls-certs\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.975547 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.975415    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-dshm\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.975677 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.975656    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-home\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.975771 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.975746    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-tmp-dir\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.975890 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.975825    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-model-cache\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.977702 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.977661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-dshm\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.977854 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.977840    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-tls-certs\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:01.985396 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:01.985370    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22snp\" (UniqueName: \"kubernetes.io/projected/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-kube-api-access-22snp\") pod \"test-model-425b2c-kserve-647c98d649-bzmk7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") " pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:02.136370 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:02.136280    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:02.407973 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:02.407949    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"]
Sep 08 11:45:02.409803 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:45:02.409781    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabdd1645_2e04_4f98_8009_1e57b3c2c5b7.slice/crio-bd0d9c6ef9f3f3e36ee105594dc84e939bc2b68f2255a921c521c7cf01fc7cca WatchSource:0}: Error finding container bd0d9c6ef9f3f3e36ee105594dc84e939bc2b68f2255a921c521c7cf01fc7cca: Status 404 returned error can't find the container with id bd0d9c6ef9f3f3e36ee105594dc84e939bc2b68f2255a921c521c7cf01fc7cca
Sep 08 11:45:02.755548 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:02.755504    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7" event={"ID":"abdd1645-2e04-4f98-8009-1e57b3c2c5b7","Type":"ContainerStarted","Data":"3db413e39d92ae14631caf4a6d2b63a81959b6eb98fe1809cf916b8b16c0163f"}
Sep 08 11:45:02.755995 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:02.755568    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7" event={"ID":"abdd1645-2e04-4f98-8009-1e57b3c2c5b7","Type":"ContainerStarted","Data":"bd0d9c6ef9f3f3e36ee105594dc84e939bc2b68f2255a921c521c7cf01fc7cca"}
Sep 08 11:45:02.778331 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:02.778192    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7" podStartSLOduration=1.778169807 podStartE2EDuration="1.778169807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:45:02.776904608 +0000 UTC m=+909.343665150" watchObservedRunningTime="2026-09-08 11:45:02.778169807 +0000 UTC m=+909.344930349"
Sep 08 11:45:02.817926 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:02.817892    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:02.897128 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:02.897100    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-425b2c-29814465-k29ht"
Sep 08 11:45:02.913572 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:02.913547    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814465-g8hnd"
Sep 08 11:45:02.923656 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:02.923612    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:03.141810 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.141685    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:03.240665 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.240635    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:03.474805 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.474771    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:03.549911 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.549870    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"]
Sep 08 11:45:03.550274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.550257    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6188b923-04db-43c4-b39f-970012e22779" containerName="cleanup"
Sep 08 11:45:03.550363 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.550277    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6188b923-04db-43c4-b39f-970012e22779" containerName="cleanup"
Sep 08 11:45:03.550363 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.550302    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75ee979-910d-4bb8-bf75-f3dba2416c46" containerName="cleanup"
Sep 08 11:45:03.550363 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.550311    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75ee979-910d-4bb8-bf75-f3dba2416c46" containerName="cleanup"
Sep 08 11:45:03.550526 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.550418    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75ee979-910d-4bb8-bf75-f3dba2416c46" containerName="cleanup"
Sep 08 11:45:03.550526 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.550431    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6188b923-04db-43c4-b39f-970012e22779" containerName="cleanup"
Sep 08 11:45:03.552940 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.552919    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"
Sep 08 11:45:03.555457 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.555438    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-46564b9d\""
Sep 08 11:45:03.565354 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.565332    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"]
Sep 08 11:45:03.568517 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.568500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"
Sep 08 11:45:03.571140 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.571117    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"]
Sep 08 11:45:03.571234 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.571117    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-46564b9d-dockercfg-256l9\""
Sep 08 11:45:03.572344 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.572317    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:03.582609 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.582589    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"]
Sep 08 11:45:03.588553 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.588530    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31d054f-f79a-4190-a11d-7eab9c11cffc-plugins-config-volume\") pod \"payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q\" (UID: \"b31d054f-f79a-4190-a11d-7eab9c11cffc\") " pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"
Sep 08 11:45:03.588682 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.588569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2rn\" (UniqueName: \"kubernetes.io/projected/c317875d-0b76-44ba-bb98-13b15f18041b-kube-api-access-wh2rn\") pod \"payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm\" (UID: \"c317875d-0b76-44ba-bb98-13b15f18041b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"
Sep 08 11:45:03.588682 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.588604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c317875d-0b76-44ba-bb98-13b15f18041b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm\" (UID: \"c317875d-0b76-44ba-bb98-13b15f18041b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"
Sep 08 11:45:03.588682 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.588632    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llsfj\" (UniqueName: \"kubernetes.io/projected/b31d054f-f79a-4190-a11d-7eab9c11cffc-kube-api-access-llsfj\") pod \"payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q\" (UID: \"b31d054f-f79a-4190-a11d-7eab9c11cffc\") " pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"
Sep 08 11:45:03.649523 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.649489    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:03.689682 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.689653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c317875d-0b76-44ba-bb98-13b15f18041b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm\" (UID: \"c317875d-0b76-44ba-bb98-13b15f18041b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"
Sep 08 11:45:03.689843 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.689689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llsfj\" (UniqueName: \"kubernetes.io/projected/b31d054f-f79a-4190-a11d-7eab9c11cffc-kube-api-access-llsfj\") pod \"payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q\" (UID: \"b31d054f-f79a-4190-a11d-7eab9c11cffc\") " pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"
Sep 08 11:45:03.689843 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.689740    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31d054f-f79a-4190-a11d-7eab9c11cffc-plugins-config-volume\") pod \"payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q\" (UID: \"b31d054f-f79a-4190-a11d-7eab9c11cffc\") " pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"
Sep 08 11:45:03.689843 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.689769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2rn\" (UniqueName: \"kubernetes.io/projected/c317875d-0b76-44ba-bb98-13b15f18041b-kube-api-access-wh2rn\") pod \"payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm\" (UID: \"c317875d-0b76-44ba-bb98-13b15f18041b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"
Sep 08 11:45:03.690393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.690375    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31d054f-f79a-4190-a11d-7eab9c11cffc-plugins-config-volume\") pod \"payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q\" (UID: \"b31d054f-f79a-4190-a11d-7eab9c11cffc\") " pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"
Sep 08 11:45:03.690433 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.690375    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c317875d-0b76-44ba-bb98-13b15f18041b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm\" (UID: \"c317875d-0b76-44ba-bb98-13b15f18041b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"
Sep 08 11:45:03.699298 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.699266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llsfj\" (UniqueName: \"kubernetes.io/projected/b31d054f-f79a-4190-a11d-7eab9c11cffc-kube-api-access-llsfj\") pod \"payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q\" (UID: \"b31d054f-f79a-4190-a11d-7eab9c11cffc\") " pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"
Sep 08 11:45:03.699402 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.699324    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2rn\" (UniqueName: \"kubernetes.io/projected/c317875d-0b76-44ba-bb98-13b15f18041b-kube-api-access-wh2rn\") pod \"payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm\" (UID: \"c317875d-0b76-44ba-bb98-13b15f18041b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"
Sep 08 11:45:03.764302 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.764221    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-425b2c-29814465-k29ht"
Sep 08 11:45:03.764302 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.764227    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-425b2c-29814465-k29ht" event={"ID":"a75ee979-910d-4bb8-bf75-f3dba2416c46","Type":"ContainerDied","Data":"c21932a1f816d61cf389950305d532b86c0f74e0ad339b819f4a8075626de17f"}
Sep 08 11:45:03.764302 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.764275    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c21932a1f816d61cf389950305d532b86c0f74e0ad339b819f4a8075626de17f"
Sep 08 11:45:03.765729 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.765702    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814465-g8hnd"
Sep 08 11:45:03.765833 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.765746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814465-g8hnd" event={"ID":"6188b923-04db-43c4-b39f-970012e22779","Type":"ContainerDied","Data":"acb74755d481b261321c9bdae1a4d5b1d44f46ba357634fca0d9584a424c3725"}
Sep 08 11:45:03.765833 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.765789    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acb74755d481b261321c9bdae1a4d5b1d44f46ba357634fca0d9584a424c3725"
Sep 08 11:45:03.863806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.863770    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"
Sep 08 11:45:03.880134 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:03.880106    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"
Sep 08 11:45:04.021978 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:04.021895    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:04.122012 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:04.121973    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:04.144712 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:04.144683    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"]
Sep 08 11:45:04.147718 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:45:04.147686    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc317875d_0b76_44ba_bb98_13b15f18041b.slice/crio-9860ab8cebfa9db25922f9a0976f0efb895f55774937834099f1f7f42b9fc281 WatchSource:0}: Error finding container 9860ab8cebfa9db25922f9a0976f0efb895f55774937834099f1f7f42b9fc281: Status 404 returned error can't find the container with id 9860ab8cebfa9db25922f9a0976f0efb895f55774937834099f1f7f42b9fc281
Sep 08 11:45:04.164982 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:04.164954    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"]
Sep 08 11:45:04.167500 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:45:04.167475    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31d054f_f79a_4190_a11d_7eab9c11cffc.slice/crio-a1f059badaa7b6e972a74cbaa70aaac34252f2c02b4be0836b2324af332e3632 WatchSource:0}: Error finding container a1f059badaa7b6e972a74cbaa70aaac34252f2c02b4be0836b2324af332e3632: Status 404 returned error can't find the container with id a1f059badaa7b6e972a74cbaa70aaac34252f2c02b4be0836b2324af332e3632
Sep 08 11:45:04.774391 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:04.774344    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q" event={"ID":"b31d054f-f79a-4190-a11d-7eab9c11cffc","Type":"ContainerStarted","Data":"53d0a24763f42b4baf05f260588ef5733d90129a9ed513396676b8a53d73d638"}
Sep 08 11:45:04.774958 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:04.774407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q" event={"ID":"b31d054f-f79a-4190-a11d-7eab9c11cffc","Type":"ContainerStarted","Data":"a1f059badaa7b6e972a74cbaa70aaac34252f2c02b4be0836b2324af332e3632"}
Sep 08 11:45:04.774958 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:04.774432    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"
Sep 08 11:45:04.776074 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:04.776048    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm" event={"ID":"c317875d-0b76-44ba-bb98-13b15f18041b","Type":"ContainerStarted","Data":"2344507d8f0a4ec8c41e60fc07ea01b6b2c535609541f3975d1f23aaf512229f"}
Sep 08 11:45:04.776189 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:04.776090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm" event={"ID":"c317875d-0b76-44ba-bb98-13b15f18041b","Type":"ContainerStarted","Data":"9860ab8cebfa9db25922f9a0976f0efb895f55774937834099f1f7f42b9fc281"}
Sep 08 11:45:04.776259 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:04.776200    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"
Sep 08 11:45:04.795208 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:04.795156    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q" podStartSLOduration=1.7951425589999999 podStartE2EDuration="1.795142559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:45:04.792996286 +0000 UTC m=+911.359756832" watchObservedRunningTime="2026-09-08 11:45:04.795142559 +0000 UTC m=+911.361903099"
Sep 08 11:45:04.810954 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:04.810907    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm" podStartSLOduration=1.810892286 podStartE2EDuration="1.810892286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:45:04.81021227 +0000 UTC m=+911.376972811" watchObservedRunningTime="2026-09-08 11:45:04.810892286 +0000 UTC m=+911.377652870"
Sep 08 11:45:07.261647 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:07.261609    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:07.349218 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:07.349190    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:08.444249 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:08.444217    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:08.546571 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:08.546537    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:08.729624 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:08.729585    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:09.165514 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.165388    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:09.258452 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.258410    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:09.259272 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.259252    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"]
Sep 08 11:45:09.262861 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.262845    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"
Sep 08 11:45:09.265612 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.265592    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-20cf96-dockercfg-m6qwf\""
Sep 08 11:45:09.265612 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.265608    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-20cf96\""
Sep 08 11:45:09.272769 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.272748    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"]
Sep 08 11:45:09.341159 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.341125    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14df874c-c7c3-439e-91b2-ab12a484f63c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx\" (UID: \"14df874c-c7c3-439e-91b2-ab12a484f63c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"
Sep 08 11:45:09.341339 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.341172    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjn42\" (UniqueName: \"kubernetes.io/projected/14df874c-c7c3-439e-91b2-ab12a484f63c-kube-api-access-kjn42\") pod \"payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx\" (UID: \"14df874c-c7c3-439e-91b2-ab12a484f63c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"
Sep 08 11:45:09.442286 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.442197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjn42\" (UniqueName: \"kubernetes.io/projected/14df874c-c7c3-439e-91b2-ab12a484f63c-kube-api-access-kjn42\") pod \"payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx\" (UID: \"14df874c-c7c3-439e-91b2-ab12a484f63c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"
Sep 08 11:45:09.442454 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.442341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14df874c-c7c3-439e-91b2-ab12a484f63c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx\" (UID: \"14df874c-c7c3-439e-91b2-ab12a484f63c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"
Sep 08 11:45:09.443045 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.443021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14df874c-c7c3-439e-91b2-ab12a484f63c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx\" (UID: \"14df874c-c7c3-439e-91b2-ab12a484f63c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"
Sep 08 11:45:09.451129 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.451103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjn42\" (UniqueName: \"kubernetes.io/projected/14df874c-c7c3-439e-91b2-ab12a484f63c-kube-api-access-kjn42\") pod \"payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx\" (UID: \"14df874c-c7c3-439e-91b2-ab12a484f63c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"
Sep 08 11:45:09.572915 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.572878    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"
Sep 08 11:45:09.849946 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:09.849919    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"]
Sep 08 11:45:09.852226 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:45:09.852199    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14df874c_c7c3_439e_91b2_ab12a484f63c.slice/crio-be2903cc958b2fc0d34b68a2e83daea528a46f03294baa6deaf52d21f681b802 WatchSource:0}: Error finding container be2903cc958b2fc0d34b68a2e83daea528a46f03294baa6deaf52d21f681b802: Status 404 returned error can't find the container with id be2903cc958b2fc0d34b68a2e83daea528a46f03294baa6deaf52d21f681b802
Sep 08 11:45:10.808025 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:10.807990    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx" event={"ID":"14df874c-c7c3-439e-91b2-ab12a484f63c","Type":"ContainerStarted","Data":"0805a1bae58c82cd5963b7ba1bddbbbf53c3625ffebe40a79262bf4ee45072a5"}
Sep 08 11:45:10.808416 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:10.808043    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx" event={"ID":"14df874c-c7c3-439e-91b2-ab12a484f63c","Type":"ContainerStarted","Data":"be2903cc958b2fc0d34b68a2e83daea528a46f03294baa6deaf52d21f681b802"}
Sep 08 11:45:10.808416 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:10.808155    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"
Sep 08 11:45:10.826738 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:10.826693    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx" podStartSLOduration=1.8266792139999999 podStartE2EDuration="1.826679214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:45:10.825931225 +0000 UTC m=+917.392691766" watchObservedRunningTime="2026-09-08 11:45:10.826679214 +0000 UTC m=+917.393439780"
Sep 08 11:45:12.136714 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:12.136676    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:12.137194 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:12.136729    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:12.152529 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:12.152502    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:12.829900 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:12.829872    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:45:15.783442 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:15.783411    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"
Sep 08 11:45:15.783917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:15.783511    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"
Sep 08 11:45:21.816242 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:21.816213    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"
Sep 08 11:45:23.898285 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:23.898249    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:26.353775 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:26.353733    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj"]
Sep 08 11:45:26.357818 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:26.357793    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj"
Sep 08 11:45:26.361220 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:26.361179    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:45:26.366669 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:26.366640    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:45:26.372942 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:26.372918    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj"]
Sep 08 11:45:26.489860 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:26.489819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d67aa29-e3d4-45fe-96a5-7ee3c177f836-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-46564b9d-86vhj\" (UID: \"7d67aa29-e3d4-45fe-96a5-7ee3c177f836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj"
Sep 08 11:45:26.591353 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:26.591315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d67aa29-e3d4-45fe-96a5-7ee3c177f836-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-46564b9d-86vhj\" (UID: \"7d67aa29-e3d4-45fe-96a5-7ee3c177f836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj"
Sep 08 11:45:26.591978 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:26.591957    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d67aa29-e3d4-45fe-96a5-7ee3c177f836-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-46564b9d-86vhj\" (UID: \"7d67aa29-e3d4-45fe-96a5-7ee3c177f836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj"
Sep 08 11:45:26.675576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:26.675488    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj"
Sep 08 11:45:26.946943 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:26.946920    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj"]
Sep 08 11:45:26.948908 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:45:26.948880    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d67aa29_e3d4_45fe_96a5_7ee3c177f836.slice/crio-d977d028cac7905558bb8cbe0819162396eb8fc444c57372cae27c7a7587d191 WatchSource:0}: Error finding container d977d028cac7905558bb8cbe0819162396eb8fc444c57372cae27c7a7587d191: Status 404 returned error can't find the container with id d977d028cac7905558bb8cbe0819162396eb8fc444c57372cae27c7a7587d191
Sep 08 11:45:27.897403 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:27.897313    2561 generic.go:356] "Generic (PLEG): container finished" podID="7d67aa29-e3d4-45fe-96a5-7ee3c177f836" containerID="d5453dc7e1ad3455edcfa1f0f2aab90445f0aa797f055a572d23762f277863c9" exitCode=0
Sep 08 11:45:27.897403 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:27.897369    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj" event={"ID":"7d67aa29-e3d4-45fe-96a5-7ee3c177f836","Type":"ContainerDied","Data":"d5453dc7e1ad3455edcfa1f0f2aab90445f0aa797f055a572d23762f277863c9"}
Sep 08 11:45:27.897865 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:27.897421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj" event={"ID":"7d67aa29-e3d4-45fe-96a5-7ee3c177f836","Type":"ContainerStarted","Data":"d977d028cac7905558bb8cbe0819162396eb8fc444c57372cae27c7a7587d191"}
Sep 08 11:45:29.032662 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:29.032633    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj"
Sep 08 11:45:29.113989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:29.113955    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d67aa29-e3d4-45fe-96a5-7ee3c177f836-maas-api-service-ca\") pod \"7d67aa29-e3d4-45fe-96a5-7ee3c177f836\" (UID: \"7d67aa29-e3d4-45fe-96a5-7ee3c177f836\") "
Sep 08 11:45:29.114325 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:29.114300    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d67aa29-e3d4-45fe-96a5-7ee3c177f836-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7d67aa29-e3d4-45fe-96a5-7ee3c177f836" (UID: "7d67aa29-e3d4-45fe-96a5-7ee3c177f836"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:45:29.215404 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:29.215372    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d67aa29-e3d4-45fe-96a5-7ee3c177f836-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:45:29.908559 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:29.908531    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj"
Sep 08 11:45:29.908760 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:29.908528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj" event={"ID":"7d67aa29-e3d4-45fe-96a5-7ee3c177f836","Type":"ContainerDied","Data":"d977d028cac7905558bb8cbe0819162396eb8fc444c57372cae27c7a7587d191"}
Sep 08 11:45:29.908760 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:29.908663    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d977d028cac7905558bb8cbe0819162396eb8fc444c57372cae27c7a7587d191"
Sep 08 11:45:30.692728 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:30.692695    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:32.164533 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:32.164439    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:33.029546 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.029512    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:33.669499 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.669451    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:33.695644 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.695610    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"]
Sep 08 11:45:33.695877 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.695851    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q" podUID="b31d054f-f79a-4190-a11d-7eab9c11cffc" containerName="payload-processing" containerID="cri-o://53d0a24763f42b4baf05f260588ef5733d90129a9ed513396676b8a53d73d638" gracePeriod=30
Sep 08 11:45:33.701339 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.701310    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"]
Sep 08 11:45:33.701596 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.701569    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm" podUID="c317875d-0b76-44ba-bb98-13b15f18041b" containerName="payload-pre-processing" containerID="cri-o://2344507d8f0a4ec8c41e60fc07ea01b6b2c535609541f3975d1f23aaf512229f" gracePeriod=30
Sep 08 11:45:33.740317 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.740279    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:33.801367 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.801335    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:33.876706 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.876673    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj"]
Sep 08 11:45:33.879183 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.879157    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-46564b9d-86vhj"]
Sep 08 11:45:33.879698 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.879675    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:33.934979 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.934950    2561 generic.go:356] "Generic (PLEG): container finished" podID="b31d054f-f79a-4190-a11d-7eab9c11cffc" containerID="53d0a24763f42b4baf05f260588ef5733d90129a9ed513396676b8a53d73d638" exitCode=0
Sep 08 11:45:33.935109 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.935022    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q" event={"ID":"b31d054f-f79a-4190-a11d-7eab9c11cffc","Type":"ContainerDied","Data":"53d0a24763f42b4baf05f260588ef5733d90129a9ed513396676b8a53d73d638"}
Sep 08 11:45:33.935109 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.935076    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q" event={"ID":"b31d054f-f79a-4190-a11d-7eab9c11cffc","Type":"ContainerDied","Data":"a1f059badaa7b6e972a74cbaa70aaac34252f2c02b4be0836b2324af332e3632"}
Sep 08 11:45:33.935109 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.935096    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1f059badaa7b6e972a74cbaa70aaac34252f2c02b4be0836b2324af332e3632"
Sep 08 11:45:33.936858 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.936820    2561 generic.go:356] "Generic (PLEG): container finished" podID="c317875d-0b76-44ba-bb98-13b15f18041b" containerID="2344507d8f0a4ec8c41e60fc07ea01b6b2c535609541f3975d1f23aaf512229f" exitCode=0
Sep 08 11:45:33.936952 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.936863    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm" event={"ID":"c317875d-0b76-44ba-bb98-13b15f18041b","Type":"ContainerDied","Data":"2344507d8f0a4ec8c41e60fc07ea01b6b2c535609541f3975d1f23aaf512229f"}
Sep 08 11:45:33.943006 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.942988    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"
Sep 08 11:45:33.947407 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.947387    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:33.965263 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.965233    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d67aa29-e3d4-45fe-96a5-7ee3c177f836" path="/var/lib/kubelet/pods/7d67aa29-e3d4-45fe-96a5-7ee3c177f836/volumes"
Sep 08 11:45:33.975378 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:33.975350    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"
Sep 08 11:45:34.051358 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.051325    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:34.061347 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.061328    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh2rn\" (UniqueName: \"kubernetes.io/projected/c317875d-0b76-44ba-bb98-13b15f18041b-kube-api-access-wh2rn\") pod \"c317875d-0b76-44ba-bb98-13b15f18041b\" (UID: \"c317875d-0b76-44ba-bb98-13b15f18041b\") "
Sep 08 11:45:34.061423 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.061381    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31d054f-f79a-4190-a11d-7eab9c11cffc-plugins-config-volume\") pod \"b31d054f-f79a-4190-a11d-7eab9c11cffc\" (UID: \"b31d054f-f79a-4190-a11d-7eab9c11cffc\") "
Sep 08 11:45:34.061462 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.061448    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c317875d-0b76-44ba-bb98-13b15f18041b-plugins-config-volume\") pod \"c317875d-0b76-44ba-bb98-13b15f18041b\" (UID: \"c317875d-0b76-44ba-bb98-13b15f18041b\") "
Sep 08 11:45:34.061540 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.061527    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llsfj\" (UniqueName: \"kubernetes.io/projected/b31d054f-f79a-4190-a11d-7eab9c11cffc-kube-api-access-llsfj\") pod \"b31d054f-f79a-4190-a11d-7eab9c11cffc\" (UID: \"b31d054f-f79a-4190-a11d-7eab9c11cffc\") "
Sep 08 11:45:34.061818 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.061787    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31d054f-f79a-4190-a11d-7eab9c11cffc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b31d054f-f79a-4190-a11d-7eab9c11cffc" (UID: "b31d054f-f79a-4190-a11d-7eab9c11cffc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:45:34.061921 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.061858    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c317875d-0b76-44ba-bb98-13b15f18041b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c317875d-0b76-44ba-bb98-13b15f18041b" (UID: "c317875d-0b76-44ba-bb98-13b15f18041b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:45:34.063648 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.063619    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31d054f-f79a-4190-a11d-7eab9c11cffc-kube-api-access-llsfj" (OuterVolumeSpecName: "kube-api-access-llsfj") pod "b31d054f-f79a-4190-a11d-7eab9c11cffc" (UID: "b31d054f-f79a-4190-a11d-7eab9c11cffc"). InnerVolumeSpecName "kube-api-access-llsfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:45:34.063733 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.063649    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c317875d-0b76-44ba-bb98-13b15f18041b-kube-api-access-wh2rn" (OuterVolumeSpecName: "kube-api-access-wh2rn") pod "c317875d-0b76-44ba-bb98-13b15f18041b" (UID: "c317875d-0b76-44ba-bb98-13b15f18041b"). InnerVolumeSpecName "kube-api-access-wh2rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:45:34.162820 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.162788    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b31d054f-f79a-4190-a11d-7eab9c11cffc-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:45:34.162820 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.162813    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c317875d-0b76-44ba-bb98-13b15f18041b-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:45:34.162820 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.162824    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llsfj\" (UniqueName: \"kubernetes.io/projected/b31d054f-f79a-4190-a11d-7eab9c11cffc-kube-api-access-llsfj\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:45:34.163049 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.162832    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh2rn\" (UniqueName: \"kubernetes.io/projected/c317875d-0b76-44ba-bb98-13b15f18041b-kube-api-access-wh2rn\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:45:34.678047 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.678013    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:34.779586 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.779551    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:34.938070 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.937987    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:34.942886 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.942863    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"
Sep 08 11:45:34.943038 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.942890    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm" event={"ID":"c317875d-0b76-44ba-bb98-13b15f18041b","Type":"ContainerDied","Data":"9860ab8cebfa9db25922f9a0976f0efb895f55774937834099f1f7f42b9fc281"}
Sep 08 11:45:34.943038 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.942949    2561 scope.go:117] "RemoveContainer" containerID="2344507d8f0a4ec8c41e60fc07ea01b6b2c535609541f3975d1f23aaf512229f"
Sep 08 11:45:34.943141 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.943091    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"
Sep 08 11:45:34.969678 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.969650    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"]
Sep 08 11:45:34.974641 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.974620    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46564b9d-685c97f454-gfzcm"]
Sep 08 11:45:34.989140 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.989115    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"]
Sep 08 11:45:34.996859 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:34.996836    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46564b9d-655c47fbd5-8d89q"]
Sep 08 11:45:35.959959 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:35.959928    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b31d054f-f79a-4190-a11d-7eab9c11cffc" path="/var/lib/kubelet/pods/b31d054f-f79a-4190-a11d-7eab9c11cffc/volumes"
Sep 08 11:45:35.960397 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:35.960261    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c317875d-0b76-44ba-bb98-13b15f18041b" path="/var/lib/kubelet/pods/c317875d-0b76-44ba-bb98-13b15f18041b/volumes"
Sep 08 11:45:36.234987 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:36.234906    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:40.240876 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:40.240838    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:42.166989 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:42.166956    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:43.078197 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:43.078165    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:43.190506 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:43.190451    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:43.280137 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:43.280103    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:43.377708 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:43.377624    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:43.800155 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:43.800117    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:49.136251 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.136200    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"]
Sep 08 11:45:49.136754 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.136709    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c317875d-0b76-44ba-bb98-13b15f18041b" containerName="payload-pre-processing"
Sep 08 11:45:49.136754 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.136725    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c317875d-0b76-44ba-bb98-13b15f18041b" containerName="payload-pre-processing"
Sep 08 11:45:49.136754 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.136751    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d67aa29-e3d4-45fe-96a5-7ee3c177f836" containerName="revoke-keys"
Sep 08 11:45:49.136937 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.136760    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d67aa29-e3d4-45fe-96a5-7ee3c177f836" containerName="revoke-keys"
Sep 08 11:45:49.136937 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.136783    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b31d054f-f79a-4190-a11d-7eab9c11cffc" containerName="payload-processing"
Sep 08 11:45:49.136937 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.136793    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b31d054f-f79a-4190-a11d-7eab9c11cffc" containerName="payload-processing"
Sep 08 11:45:49.136937 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.136866    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c317875d-0b76-44ba-bb98-13b15f18041b" containerName="payload-pre-processing"
Sep 08 11:45:49.136937 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.136880    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b31d054f-f79a-4190-a11d-7eab9c11cffc" containerName="payload-processing"
Sep 08 11:45:49.136937 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.136891    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d67aa29-e3d4-45fe-96a5-7ee3c177f836" containerName="revoke-keys"
Sep 08 11:45:49.146026 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.146004    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.148730 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.148679    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-20cf96\"/\"openshift-service-ca.crt\""
Sep 08 11:45:49.148876 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.148837    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-20cf96\"/\"kube-root-ca.crt\""
Sep 08 11:45:49.150093 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.150071    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-20cf96\"/\"default-dockercfg-vdpkv\""
Sep 08 11:45:49.150224 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.150197    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-20cf96\"/\"explicit-model-20cf96-kserve-self-signed-certs\""
Sep 08 11:45:49.150352 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.150135    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"]
Sep 08 11:45:49.171385 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.171357    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:49.203676 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.203639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-model-cache\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.203676 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.203672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j55mn\" (UniqueName: \"kubernetes.io/projected/8abe57bf-9f39-48cb-8454-bbe2977f4b24-kube-api-access-j55mn\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.203905 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.203696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-home\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.203905 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.203771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-dshm\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.203905 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.203807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-tmp-dir\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.203905 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.203851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abe57bf-9f39-48cb-8454-bbe2977f4b24-tls-certs\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.304499 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.304452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-model-cache\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.304499 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.304497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j55mn\" (UniqueName: \"kubernetes.io/projected/8abe57bf-9f39-48cb-8454-bbe2977f4b24-kube-api-access-j55mn\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.304758 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.304516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-home\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.304758 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.304554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-dshm\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.304758 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.304577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-tmp-dir\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.304758 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.304615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abe57bf-9f39-48cb-8454-bbe2977f4b24-tls-certs\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.304968 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.304909    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-model-cache\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.304968 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.304915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-home\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.305087 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.305067    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-tmp-dir\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.306836 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.306815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-dshm\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.307110 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.307092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abe57bf-9f39-48cb-8454-bbe2977f4b24-tls-certs\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.315276 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.315254    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j55mn\" (UniqueName: \"kubernetes.io/projected/8abe57bf-9f39-48cb-8454-bbe2977f4b24-kube-api-access-j55mn\") pod \"explicit-model-20cf96-kserve-5fb95556d9-4bgt4\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") " pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.457044 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.457006    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:49.735422 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:49.735395    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"]
Sep 08 11:45:49.737999 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:45:49.737971    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8abe57bf_9f39_48cb_8454_bbe2977f4b24.slice/crio-d5e5d308822e4e7b7470c77a993fa6c37388f4cd0a7ffd770f12836bf4274cd6 WatchSource:0}: Error finding container d5e5d308822e4e7b7470c77a993fa6c37388f4cd0a7ffd770f12836bf4274cd6: Status 404 returned error can't find the container with id d5e5d308822e4e7b7470c77a993fa6c37388f4cd0a7ffd770f12836bf4274cd6
Sep 08 11:45:50.024048 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:50.023956    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4" event={"ID":"8abe57bf-9f39-48cb-8454-bbe2977f4b24","Type":"ContainerStarted","Data":"b4d74e390eb54c3e19b302d3d00f6a0f608be638b5b712ba611a021654725cfc"}
Sep 08 11:45:50.024048 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:50.024001    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4" event={"ID":"8abe57bf-9f39-48cb-8454-bbe2977f4b24","Type":"ContainerStarted","Data":"d5e5d308822e4e7b7470c77a993fa6c37388f4cd0a7ffd770f12836bf4274cd6"}
Sep 08 11:45:50.043945 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:50.043894    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4" podStartSLOduration=1.043876555 podStartE2EDuration="1.043876555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:45:50.043639108 +0000 UTC m=+956.610399646" watchObservedRunningTime="2026-09-08 11:45:50.043876555 +0000 UTC m=+956.610637100"
Sep 08 11:45:54.992903 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:54.992864    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:45:59.457155 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:59.457108    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:59.457588 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:59.457175    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:45:59.469996 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:45:59.469965    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:46:00.092037 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:00.092005    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:46:08.387455 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:08.387418    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"]
Sep 08 11:46:08.455066 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:08.455032    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:08.506126 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:08.506092    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd"]
Sep 08 11:46:08.510892 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:08.510877    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd"
Sep 08 11:46:08.513775 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:08.513751    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:46:08.513775 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:08.513766    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:46:08.520084 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:08.520056    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd"]
Sep 08 11:46:08.562173 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:08.562140    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/216a8090-e587-4c85-b55d-1c288d3519b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd\" (UID: \"216a8090-e587-4c85-b55d-1c288d3519b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd"
Sep 08 11:46:08.663146 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:08.663048    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/216a8090-e587-4c85-b55d-1c288d3519b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd\" (UID: \"216a8090-e587-4c85-b55d-1c288d3519b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd"
Sep 08 11:46:08.663776 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:08.663750    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/216a8090-e587-4c85-b55d-1c288d3519b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd\" (UID: \"216a8090-e587-4c85-b55d-1c288d3519b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd"
Sep 08 11:46:08.682073 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:08.682043    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:08.822056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:08.822021    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd"
Sep 08 11:46:09.096796 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:09.096707    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd"]
Sep 08 11:46:09.099107 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:46:09.099082    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216a8090_e587_4c85_b55d_1c288d3519b8.slice/crio-9a7eb4c5b16cd99725e4b090da439fb19719fb17d22a0c8bdbf717cc2cb8ca0e WatchSource:0}: Error finding container 9a7eb4c5b16cd99725e4b090da439fb19719fb17d22a0c8bdbf717cc2cb8ca0e: Status 404 returned error can't find the container with id 9a7eb4c5b16cd99725e4b090da439fb19719fb17d22a0c8bdbf717cc2cb8ca0e
Sep 08 11:46:09.126124 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:09.126098    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd" event={"ID":"216a8090-e587-4c85-b55d-1c288d3519b8","Type":"ContainerStarted","Data":"9a7eb4c5b16cd99725e4b090da439fb19719fb17d22a0c8bdbf717cc2cb8ca0e"}
Sep 08 11:46:09.682442 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:09.682403    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:10.132877 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:10.132838    2561 generic.go:356] "Generic (PLEG): container finished" podID="216a8090-e587-4c85-b55d-1c288d3519b8" containerID="568c403a8da3c6682bbef758eb2130b637a3563a69f3158298e0d46364a183a3" exitCode=0
Sep 08 11:46:10.133064 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:10.132922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd" event={"ID":"216a8090-e587-4c85-b55d-1c288d3519b8","Type":"ContainerDied","Data":"568c403a8da3c6682bbef758eb2130b637a3563a69f3158298e0d46364a183a3"}
Sep 08 11:46:11.277081 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:11.277051    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd"
Sep 08 11:46:11.386970 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:11.386940    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/216a8090-e587-4c85-b55d-1c288d3519b8-maas-api-service-ca\") pod \"216a8090-e587-4c85-b55d-1c288d3519b8\" (UID: \"216a8090-e587-4c85-b55d-1c288d3519b8\") "
Sep 08 11:46:11.387283 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:11.387261    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216a8090-e587-4c85-b55d-1c288d3519b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "216a8090-e587-4c85-b55d-1c288d3519b8" (UID: "216a8090-e587-4c85-b55d-1c288d3519b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:46:11.488305 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:11.488264    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/216a8090-e587-4c85-b55d-1c288d3519b8-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:12.150425 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:12.150399    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd"
Sep 08 11:46:12.150425 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:12.150411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd" event={"ID":"216a8090-e587-4c85-b55d-1c288d3519b8","Type":"ContainerDied","Data":"9a7eb4c5b16cd99725e4b090da439fb19719fb17d22a0c8bdbf717cc2cb8ca0e"}
Sep 08 11:46:12.150657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:12.150452    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a7eb4c5b16cd99725e4b090da439fb19719fb17d22a0c8bdbf717cc2cb8ca0e"
Sep 08 11:46:16.670931 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:16.670853    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:16.799830 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:16.799792    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:16.897901 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:16.897852    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:19.100971 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.100922    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:19.109716 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.109676    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"]
Sep 08 11:46:19.109948 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.109921    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx" podUID="14df874c-c7c3-439e-91b2-ab12a484f63c" containerName="payload-pre-processing" containerID="cri-o://0805a1bae58c82cd5963b7ba1bddbbbf53c3625ffebe40a79262bf4ee45072a5" gracePeriod=30
Sep 08 11:46:19.199204 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.199170    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:19.300162 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.300128    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:19.373142 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.373107    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"
Sep 08 11:46:19.374341 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.374313    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd"]
Sep 08 11:46:19.376992 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.376969    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-20cf96-qj6qd"]
Sep 08 11:46:19.399551 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.399500    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:19.463605 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.463571    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14df874c-c7c3-439e-91b2-ab12a484f63c-plugins-config-volume\") pod \"14df874c-c7c3-439e-91b2-ab12a484f63c\" (UID: \"14df874c-c7c3-439e-91b2-ab12a484f63c\") "
Sep 08 11:46:19.463605 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.463606    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjn42\" (UniqueName: \"kubernetes.io/projected/14df874c-c7c3-439e-91b2-ab12a484f63c-kube-api-access-kjn42\") pod \"14df874c-c7c3-439e-91b2-ab12a484f63c\" (UID: \"14df874c-c7c3-439e-91b2-ab12a484f63c\") "
Sep 08 11:46:19.463957 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.463932    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14df874c-c7c3-439e-91b2-ab12a484f63c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14df874c-c7c3-439e-91b2-ab12a484f63c" (UID: "14df874c-c7c3-439e-91b2-ab12a484f63c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:46:19.465717 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.465694    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14df874c-c7c3-439e-91b2-ab12a484f63c-kube-api-access-kjn42" (OuterVolumeSpecName: "kube-api-access-kjn42") pod "14df874c-c7c3-439e-91b2-ab12a484f63c" (UID: "14df874c-c7c3-439e-91b2-ab12a484f63c"). InnerVolumeSpecName "kube-api-access-kjn42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:46:19.564438 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.564394    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14df874c-c7c3-439e-91b2-ab12a484f63c-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:19.564438 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.564432    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjn42\" (UniqueName: \"kubernetes.io/projected/14df874c-c7c3-439e-91b2-ab12a484f63c-kube-api-access-kjn42\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:19.752267 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.752235    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:19.961184 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:19.961152    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="216a8090-e587-4c85-b55d-1c288d3519b8" path="/var/lib/kubelet/pods/216a8090-e587-4c85-b55d-1c288d3519b8/volumes"
Sep 08 11:46:20.192328 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:20.192236    2561 generic.go:356] "Generic (PLEG): container finished" podID="14df874c-c7c3-439e-91b2-ab12a484f63c" containerID="0805a1bae58c82cd5963b7ba1bddbbbf53c3625ffebe40a79262bf4ee45072a5" exitCode=0
Sep 08 11:46:20.192328 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:20.192280    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"
Sep 08 11:46:20.192328 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:20.192319    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx" event={"ID":"14df874c-c7c3-439e-91b2-ab12a484f63c","Type":"ContainerDied","Data":"0805a1bae58c82cd5963b7ba1bddbbbf53c3625ffebe40a79262bf4ee45072a5"}
Sep 08 11:46:20.192845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:20.192370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx" event={"ID":"14df874c-c7c3-439e-91b2-ab12a484f63c","Type":"ContainerDied","Data":"be2903cc958b2fc0d34b68a2e83daea528a46f03294baa6deaf52d21f681b802"}
Sep 08 11:46:20.192845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:20.192394    2561 scope.go:117] "RemoveContainer" containerID="0805a1bae58c82cd5963b7ba1bddbbbf53c3625ffebe40a79262bf4ee45072a5"
Sep 08 11:46:20.201148 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:20.201128    2561 scope.go:117] "RemoveContainer" containerID="0805a1bae58c82cd5963b7ba1bddbbbf53c3625ffebe40a79262bf4ee45072a5"
Sep 08 11:46:20.201421 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:46:20.201392    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0805a1bae58c82cd5963b7ba1bddbbbf53c3625ffebe40a79262bf4ee45072a5\": container with ID starting with 0805a1bae58c82cd5963b7ba1bddbbbf53c3625ffebe40a79262bf4ee45072a5 not found: ID does not exist" containerID="0805a1bae58c82cd5963b7ba1bddbbbf53c3625ffebe40a79262bf4ee45072a5"
Sep 08 11:46:20.201590 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:20.201421    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0805a1bae58c82cd5963b7ba1bddbbbf53c3625ffebe40a79262bf4ee45072a5"} err="failed to get container status \"0805a1bae58c82cd5963b7ba1bddbbbf53c3625ffebe40a79262bf4ee45072a5\": rpc error: code = NotFound desc = could not find container \"0805a1bae58c82cd5963b7ba1bddbbbf53c3625ffebe40a79262bf4ee45072a5\": container with ID starting with 0805a1bae58c82cd5963b7ba1bddbbbf53c3625ffebe40a79262bf4ee45072a5 not found: ID does not exist"
Sep 08 11:46:20.211832 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:20.211808    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"]
Sep 08 11:46:20.216884 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:20.216864    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-20cf96-786b67c454-9qstx"]
Sep 08 11:46:21.960770 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:21.960736    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14df874c-c7c3-439e-91b2-ab12a484f63c" path="/var/lib/kubelet/pods/14df874c-c7c3-439e-91b2-ab12a484f63c/volumes"
Sep 08 11:46:23.406822 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.406783    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4" podUID="8abe57bf-9f39-48cb-8454-bbe2977f4b24" containerName="main" containerID="cri-o://b4d74e390eb54c3e19b302d3d00f6a0f608be638b5b712ba611a021654725cfc" gracePeriod=45
Sep 08 11:46:23.651109 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.651085    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:46:23.800935 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.800898    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-model-cache\") pod \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") "
Sep 08 11:46:23.801121 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.801044    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j55mn\" (UniqueName: \"kubernetes.io/projected/8abe57bf-9f39-48cb-8454-bbe2977f4b24-kube-api-access-j55mn\") pod \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") "
Sep 08 11:46:23.801121 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.801078    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abe57bf-9f39-48cb-8454-bbe2977f4b24-tls-certs\") pod \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") "
Sep 08 11:46:23.801262 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.801123    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-home\") pod \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") "
Sep 08 11:46:23.801262 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.801166    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-model-cache" (OuterVolumeSpecName: "model-cache") pod "8abe57bf-9f39-48cb-8454-bbe2977f4b24" (UID: "8abe57bf-9f39-48cb-8454-bbe2977f4b24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:46:23.801262 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.801240    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-dshm\") pod \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") "
Sep 08 11:46:23.801419 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.801272    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-tmp-dir\") pod \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\" (UID: \"8abe57bf-9f39-48cb-8454-bbe2977f4b24\") "
Sep 08 11:46:23.801419 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.801341    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-home" (OuterVolumeSpecName: "home") pod "8abe57bf-9f39-48cb-8454-bbe2977f4b24" (UID: "8abe57bf-9f39-48cb-8454-bbe2977f4b24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:46:23.801570 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.801551    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-home\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:23.801626 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.801569    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-model-cache\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:23.801626 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.801552    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8abe57bf-9f39-48cb-8454-bbe2977f4b24" (UID: "8abe57bf-9f39-48cb-8454-bbe2977f4b24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:46:23.803358 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.803333    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-dshm" (OuterVolumeSpecName: "dshm") pod "8abe57bf-9f39-48cb-8454-bbe2977f4b24" (UID: "8abe57bf-9f39-48cb-8454-bbe2977f4b24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:46:23.803479 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.803341    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8abe57bf-9f39-48cb-8454-bbe2977f4b24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8abe57bf-9f39-48cb-8454-bbe2977f4b24" (UID: "8abe57bf-9f39-48cb-8454-bbe2977f4b24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:46:23.803479 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.803356    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abe57bf-9f39-48cb-8454-bbe2977f4b24-kube-api-access-j55mn" (OuterVolumeSpecName: "kube-api-access-j55mn") pod "8abe57bf-9f39-48cb-8454-bbe2977f4b24" (UID: "8abe57bf-9f39-48cb-8454-bbe2977f4b24"). InnerVolumeSpecName "kube-api-access-j55mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:46:23.902934 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.902896    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j55mn\" (UniqueName: \"kubernetes.io/projected/8abe57bf-9f39-48cb-8454-bbe2977f4b24-kube-api-access-j55mn\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:23.902934 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.902929    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abe57bf-9f39-48cb-8454-bbe2977f4b24-tls-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:23.902934 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.902939    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-dshm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:23.903160 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:23.902947    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abe57bf-9f39-48cb-8454-bbe2977f4b24-tmp-dir\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:24.218770 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:24.218735    2561 generic.go:356] "Generic (PLEG): container finished" podID="8abe57bf-9f39-48cb-8454-bbe2977f4b24" containerID="b4d74e390eb54c3e19b302d3d00f6a0f608be638b5b712ba611a021654725cfc" exitCode=0
Sep 08 11:46:24.218983 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:24.218773    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4" event={"ID":"8abe57bf-9f39-48cb-8454-bbe2977f4b24","Type":"ContainerDied","Data":"b4d74e390eb54c3e19b302d3d00f6a0f608be638b5b712ba611a021654725cfc"}
Sep 08 11:46:24.218983 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:24.218805    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"
Sep 08 11:46:24.218983 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:24.218819    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4" event={"ID":"8abe57bf-9f39-48cb-8454-bbe2977f4b24","Type":"ContainerDied","Data":"d5e5d308822e4e7b7470c77a993fa6c37388f4cd0a7ffd770f12836bf4274cd6"}
Sep 08 11:46:24.218983 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:24.218840    2561 scope.go:117] "RemoveContainer" containerID="b4d74e390eb54c3e19b302d3d00f6a0f608be638b5b712ba611a021654725cfc"
Sep 08 11:46:24.227785 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:24.227768    2561 scope.go:117] "RemoveContainer" containerID="b4d74e390eb54c3e19b302d3d00f6a0f608be638b5b712ba611a021654725cfc"
Sep 08 11:46:24.228059 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:46:24.228042    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d74e390eb54c3e19b302d3d00f6a0f608be638b5b712ba611a021654725cfc\": container with ID starting with b4d74e390eb54c3e19b302d3d00f6a0f608be638b5b712ba611a021654725cfc not found: ID does not exist" containerID="b4d74e390eb54c3e19b302d3d00f6a0f608be638b5b712ba611a021654725cfc"
Sep 08 11:46:24.228128 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:24.228066    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d74e390eb54c3e19b302d3d00f6a0f608be638b5b712ba611a021654725cfc"} err="failed to get container status \"b4d74e390eb54c3e19b302d3d00f6a0f608be638b5b712ba611a021654725cfc\": rpc error: code = NotFound desc = could not find container \"b4d74e390eb54c3e19b302d3d00f6a0f608be638b5b712ba611a021654725cfc\": container with ID starting with b4d74e390eb54c3e19b302d3d00f6a0f608be638b5b712ba611a021654725cfc not found: ID does not exist"
Sep 08 11:46:24.238633 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:24.238578    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"]
Sep 08 11:46:24.240574 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:24.240552    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-20cf96/explicit-model-20cf96-kserve-5fb95556d9-4bgt4"]
Sep 08 11:46:24.833341 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:24.833311    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"]
Sep 08 11:46:24.941279 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:24.941232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:25.961021 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:25.960990    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8abe57bf-9f39-48cb-8454-bbe2977f4b24" path="/var/lib/kubelet/pods/8abe57bf-9f39-48cb-8454-bbe2977f4b24/volumes"
Sep 08 11:46:26.324885 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.324800    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"]
Sep 08 11:46:26.355840 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.355809    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:26.451011 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.450979    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr"]
Sep 08 11:46:26.451327 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.451314    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14df874c-c7c3-439e-91b2-ab12a484f63c" containerName="payload-pre-processing"
Sep 08 11:46:26.451327 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.451327    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="14df874c-c7c3-439e-91b2-ab12a484f63c" containerName="payload-pre-processing"
Sep 08 11:46:26.451444 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.451335    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8abe57bf-9f39-48cb-8454-bbe2977f4b24" containerName="main"
Sep 08 11:46:26.451444 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.451340    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8abe57bf-9f39-48cb-8454-bbe2977f4b24" containerName="main"
Sep 08 11:46:26.451444 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.451354    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="216a8090-e587-4c85-b55d-1c288d3519b8" containerName="revoke-keys"
Sep 08 11:46:26.451444 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.451360    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="216a8090-e587-4c85-b55d-1c288d3519b8" containerName="revoke-keys"
Sep 08 11:46:26.451444 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.451410    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="14df874c-c7c3-439e-91b2-ab12a484f63c" containerName="payload-pre-processing"
Sep 08 11:46:26.451444 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.451417    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8abe57bf-9f39-48cb-8454-bbe2977f4b24" containerName="main"
Sep 08 11:46:26.451444 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.451425    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="216a8090-e587-4c85-b55d-1c288d3519b8" containerName="revoke-keys"
Sep 08 11:46:26.455587 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.455567    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr"
Sep 08 11:46:26.459785 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.459762    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:46:26.459785 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.459777    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:46:26.470304 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.470279    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr"]
Sep 08 11:46:26.524868 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.524831    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff4118c-3392-42da-b846-7a7b2b471cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr\" (UID: \"6ff4118c-3392-42da-b846-7a7b2b471cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr"
Sep 08 11:46:26.606693 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.606591    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:26.625868 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.625833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff4118c-3392-42da-b846-7a7b2b471cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr\" (UID: \"6ff4118c-3392-42da-b846-7a7b2b471cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr"
Sep 08 11:46:26.626519 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.626497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff4118c-3392-42da-b846-7a7b2b471cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr\" (UID: \"6ff4118c-3392-42da-b846-7a7b2b471cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr"
Sep 08 11:46:26.768960 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:26.768922    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr"
Sep 08 11:46:27.042193 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:27.042165    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr"]
Sep 08 11:46:27.044166 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:46:27.044137    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff4118c_3392_42da_b846_7a7b2b471cc0.slice/crio-bbe0d80dd69ff50ff9e003b59b52927ded76c4058996297525da833b50082304 WatchSource:0}: Error finding container bbe0d80dd69ff50ff9e003b59b52927ded76c4058996297525da833b50082304: Status 404 returned error can't find the container with id bbe0d80dd69ff50ff9e003b59b52927ded76c4058996297525da833b50082304
Sep 08 11:46:27.238394 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:27.238359    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr" event={"ID":"6ff4118c-3392-42da-b846-7a7b2b471cc0","Type":"ContainerStarted","Data":"bbe0d80dd69ff50ff9e003b59b52927ded76c4058996297525da833b50082304"}
Sep 08 11:46:27.358162 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:27.358136    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:27.464538 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:27.464505    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:27.668171 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:27.668079    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:28.095992 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:28.095955    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:28.244732 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:28.244700    2561 generic.go:356] "Generic (PLEG): container finished" podID="6ff4118c-3392-42da-b846-7a7b2b471cc0" containerID="bab788a52e68f29f648e70b2e18df7c7c0b22e15f6e08eeb4aa0996655ef5d5d" exitCode=0
Sep 08 11:46:28.244902 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:28.244789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr" event={"ID":"6ff4118c-3392-42da-b846-7a7b2b471cc0","Type":"ContainerDied","Data":"bab788a52e68f29f648e70b2e18df7c7c0b22e15f6e08eeb4aa0996655ef5d5d"}
Sep 08 11:46:28.425540 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:28.425440    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:28.744437 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:28.744404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:29.376302 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:29.376279    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr"
Sep 08 11:46:29.450323 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:29.450293    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff4118c-3392-42da-b846-7a7b2b471cc0-maas-api-service-ca\") pod \"6ff4118c-3392-42da-b846-7a7b2b471cc0\" (UID: \"6ff4118c-3392-42da-b846-7a7b2b471cc0\") "
Sep 08 11:46:29.450670 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:29.450646    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff4118c-3392-42da-b846-7a7b2b471cc0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ff4118c-3392-42da-b846-7a7b2b471cc0" (UID: "6ff4118c-3392-42da-b846-7a7b2b471cc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:46:29.551581 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:29.551491    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff4118c-3392-42da-b846-7a7b2b471cc0-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:29.987234 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:29.987193    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:30.083766 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:30.083730    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:30.256725 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:30.256645    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr"
Sep 08 11:46:30.256725 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:30.256669    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr" event={"ID":"6ff4118c-3392-42da-b846-7a7b2b471cc0","Type":"ContainerDied","Data":"bbe0d80dd69ff50ff9e003b59b52927ded76c4058996297525da833b50082304"}
Sep 08 11:46:30.256725 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:30.256710    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbe0d80dd69ff50ff9e003b59b52927ded76c4058996297525da833b50082304"
Sep 08 11:46:30.507353 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:30.507218    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:32.722906 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:32.722864    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:36.977044 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:36.977002    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:36.990092 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:36.990062    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"]
Sep 08 11:46:36.990375 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:36.990321    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p" podUID="a631a187-cc9d-4e74-ab8c-028b6013ff02" containerName="payload-processing" containerID="cri-o://3416999acfa4c811c631103362e4fd2a1bcb3438e43d6c00516ebb2c1f660c7d" gracePeriod=30
Sep 08 11:46:36.997658 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:36.997624    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"]
Sep 08 11:46:36.998139 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:36.997852    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k" podUID="d93e1465-3497-40b4-a67f-7232c883c545" containerName="payload-pre-processing" containerID="cri-o://98a42b455d0b56d43637817f02c0525ce0152eea1d241bb6ea090268b3e4f54d" gracePeriod=30
Sep 08 11:46:37.065480 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.065432    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:37.161230 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.161181    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr"]
Sep 08 11:46:37.168219 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.168050    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:37.170862 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.170835    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-95669b-8vzxr"]
Sep 08 11:46:37.238636 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.238578    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"
Sep 08 11:46:37.264608 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.264584    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:37.270838 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.270815    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"
Sep 08 11:46:37.297741 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.297709    2561 generic.go:356] "Generic (PLEG): container finished" podID="d93e1465-3497-40b4-a67f-7232c883c545" containerID="98a42b455d0b56d43637817f02c0525ce0152eea1d241bb6ea090268b3e4f54d" exitCode=0
Sep 08 11:46:37.297893 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.297745    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"
Sep 08 11:46:37.297893 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.297785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k" event={"ID":"d93e1465-3497-40b4-a67f-7232c883c545","Type":"ContainerDied","Data":"98a42b455d0b56d43637817f02c0525ce0152eea1d241bb6ea090268b3e4f54d"}
Sep 08 11:46:37.297893 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.297839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k" event={"ID":"d93e1465-3497-40b4-a67f-7232c883c545","Type":"ContainerDied","Data":"e8c00c94df64d5bb406986465d64283195b5951a2dd5a1f1c23c8623cb6f66da"}
Sep 08 11:46:37.297893 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.297868    2561 scope.go:117] "RemoveContainer" containerID="98a42b455d0b56d43637817f02c0525ce0152eea1d241bb6ea090268b3e4f54d"
Sep 08 11:46:37.299339 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.299320    2561 generic.go:356] "Generic (PLEG): container finished" podID="a631a187-cc9d-4e74-ab8c-028b6013ff02" containerID="3416999acfa4c811c631103362e4fd2a1bcb3438e43d6c00516ebb2c1f660c7d" exitCode=0
Sep 08 11:46:37.299448 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.299370    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"
Sep 08 11:46:37.299448 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.299377    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p" event={"ID":"a631a187-cc9d-4e74-ab8c-028b6013ff02","Type":"ContainerDied","Data":"3416999acfa4c811c631103362e4fd2a1bcb3438e43d6c00516ebb2c1f660c7d"}
Sep 08 11:46:37.299448 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.299410    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p" event={"ID":"a631a187-cc9d-4e74-ab8c-028b6013ff02","Type":"ContainerDied","Data":"3ca979517c216ccbb0f00d39a96be5d9982759168e0d97d8baf7e461098e0cf2"}
Sep 08 11:46:37.307822 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.307796    2561 scope.go:117] "RemoveContainer" containerID="98a42b455d0b56d43637817f02c0525ce0152eea1d241bb6ea090268b3e4f54d"
Sep 08 11:46:37.308132 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:46:37.308112    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a42b455d0b56d43637817f02c0525ce0152eea1d241bb6ea090268b3e4f54d\": container with ID starting with 98a42b455d0b56d43637817f02c0525ce0152eea1d241bb6ea090268b3e4f54d not found: ID does not exist" containerID="98a42b455d0b56d43637817f02c0525ce0152eea1d241bb6ea090268b3e4f54d"
Sep 08 11:46:37.308205 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.308141    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a42b455d0b56d43637817f02c0525ce0152eea1d241bb6ea090268b3e4f54d"} err="failed to get container status \"98a42b455d0b56d43637817f02c0525ce0152eea1d241bb6ea090268b3e4f54d\": rpc error: code = NotFound desc = could not find container \"98a42b455d0b56d43637817f02c0525ce0152eea1d241bb6ea090268b3e4f54d\": container with ID starting with 98a42b455d0b56d43637817f02c0525ce0152eea1d241bb6ea090268b3e4f54d not found: ID does not exist"
Sep 08 11:46:37.308205 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.308158    2561 scope.go:117] "RemoveContainer" containerID="3416999acfa4c811c631103362e4fd2a1bcb3438e43d6c00516ebb2c1f660c7d"
Sep 08 11:46:37.323635 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.323599    2561 scope.go:117] "RemoveContainer" containerID="3416999acfa4c811c631103362e4fd2a1bcb3438e43d6c00516ebb2c1f660c7d"
Sep 08 11:46:37.323893 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:46:37.323872    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3416999acfa4c811c631103362e4fd2a1bcb3438e43d6c00516ebb2c1f660c7d\": container with ID starting with 3416999acfa4c811c631103362e4fd2a1bcb3438e43d6c00516ebb2c1f660c7d not found: ID does not exist" containerID="3416999acfa4c811c631103362e4fd2a1bcb3438e43d6c00516ebb2c1f660c7d"
Sep 08 11:46:37.323970 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.323905    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3416999acfa4c811c631103362e4fd2a1bcb3438e43d6c00516ebb2c1f660c7d"} err="failed to get container status \"3416999acfa4c811c631103362e4fd2a1bcb3438e43d6c00516ebb2c1f660c7d\": rpc error: code = NotFound desc = could not find container \"3416999acfa4c811c631103362e4fd2a1bcb3438e43d6c00516ebb2c1f660c7d\": container with ID starting with 3416999acfa4c811c631103362e4fd2a1bcb3438e43d6c00516ebb2c1f660c7d not found: ID does not exist"
Sep 08 11:46:37.417169 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.417126    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a631a187-cc9d-4e74-ab8c-028b6013ff02-plugins-config-volume\") pod \"a631a187-cc9d-4e74-ab8c-028b6013ff02\" (UID: \"a631a187-cc9d-4e74-ab8c-028b6013ff02\") "
Sep 08 11:46:37.417360 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.417197    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r8vn\" (UniqueName: \"kubernetes.io/projected/a631a187-cc9d-4e74-ab8c-028b6013ff02-kube-api-access-6r8vn\") pod \"a631a187-cc9d-4e74-ab8c-028b6013ff02\" (UID: \"a631a187-cc9d-4e74-ab8c-028b6013ff02\") "
Sep 08 11:46:37.417360 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.417242    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93e1465-3497-40b4-a67f-7232c883c545-plugins-config-volume\") pod \"d93e1465-3497-40b4-a67f-7232c883c545\" (UID: \"d93e1465-3497-40b4-a67f-7232c883c545\") "
Sep 08 11:46:37.417360 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.417290    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmrbr\" (UniqueName: \"kubernetes.io/projected/d93e1465-3497-40b4-a67f-7232c883c545-kube-api-access-dmrbr\") pod \"d93e1465-3497-40b4-a67f-7232c883c545\" (UID: \"d93e1465-3497-40b4-a67f-7232c883c545\") "
Sep 08 11:46:37.417595 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.417571    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a631a187-cc9d-4e74-ab8c-028b6013ff02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a631a187-cc9d-4e74-ab8c-028b6013ff02" (UID: "a631a187-cc9d-4e74-ab8c-028b6013ff02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:46:37.417673 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.417649    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93e1465-3497-40b4-a67f-7232c883c545-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d93e1465-3497-40b4-a67f-7232c883c545" (UID: "d93e1465-3497-40b4-a67f-7232c883c545"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:46:37.419423 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.419396    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93e1465-3497-40b4-a67f-7232c883c545-kube-api-access-dmrbr" (OuterVolumeSpecName: "kube-api-access-dmrbr") pod "d93e1465-3497-40b4-a67f-7232c883c545" (UID: "d93e1465-3497-40b4-a67f-7232c883c545"). InnerVolumeSpecName "kube-api-access-dmrbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:46:37.419549 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.419429    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a631a187-cc9d-4e74-ab8c-028b6013ff02-kube-api-access-6r8vn" (OuterVolumeSpecName: "kube-api-access-6r8vn") pod "a631a187-cc9d-4e74-ab8c-028b6013ff02" (UID: "a631a187-cc9d-4e74-ab8c-028b6013ff02"). InnerVolumeSpecName "kube-api-access-6r8vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:46:37.518778 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.518692    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmrbr\" (UniqueName: \"kubernetes.io/projected/d93e1465-3497-40b4-a67f-7232c883c545-kube-api-access-dmrbr\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:37.518778 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.518722    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a631a187-cc9d-4e74-ab8c-028b6013ff02-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:37.518778 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.518732    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r8vn\" (UniqueName: \"kubernetes.io/projected/a631a187-cc9d-4e74-ab8c-028b6013ff02-kube-api-access-6r8vn\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:37.518778 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.518740    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d93e1465-3497-40b4-a67f-7232c883c545-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:37.547214 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.547181    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:37.625128 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.625097    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"]
Sep 08 11:46:37.629506 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.629482    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-95669b-84f6674b87-zwl8p"]
Sep 08 11:46:37.641824 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.641799    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"]
Sep 08 11:46:37.646566 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.646543    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-95669b-b88688fc4-8gw6k"]
Sep 08 11:46:37.960733 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.960696    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ff4118c-3392-42da-b846-7a7b2b471cc0" path="/var/lib/kubelet/pods/6ff4118c-3392-42da-b846-7a7b2b471cc0/volumes"
Sep 08 11:46:37.961141 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.961121    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a631a187-cc9d-4e74-ab8c-028b6013ff02" path="/var/lib/kubelet/pods/a631a187-cc9d-4e74-ab8c-028b6013ff02/volumes"
Sep 08 11:46:37.961533 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:37.961514    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d93e1465-3497-40b4-a67f-7232c883c545" path="/var/lib/kubelet/pods/d93e1465-3497-40b4-a67f-7232c883c545/volumes"
Sep 08 11:46:39.853455 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:39.853415    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs" podUID="87e6aee8-8be7-4ca0-ac95-b001372a6b45" containerName="main" containerID="cri-o://95b9d1d46abd24d5d7780c62777c047f011cb76786758d56b308bac2b79c0b18" gracePeriod=45
Sep 08 11:46:39.980251 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:39.980202    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:40.083417 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.083342    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:40.097674 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.097645    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:46:40.177296 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.177208    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:40.240283 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.240257    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-tmp-dir\") pod \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") "
Sep 08 11:46:40.240283 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.240288    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-model-cache\") pod \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") "
Sep 08 11:46:40.240552 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.240320    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e6aee8-8be7-4ca0-ac95-b001372a6b45-tls-certs\") pod \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") "
Sep 08 11:46:40.240552 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.240353    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6bc8\" (UniqueName: \"kubernetes.io/projected/87e6aee8-8be7-4ca0-ac95-b001372a6b45-kube-api-access-r6bc8\") pod \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") "
Sep 08 11:46:40.240552 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.240406    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-home\") pod \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") "
Sep 08 11:46:40.240552 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.240431    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-dshm\") pod \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\" (UID: \"87e6aee8-8be7-4ca0-ac95-b001372a6b45\") "
Sep 08 11:46:40.240753 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.240599    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87e6aee8-8be7-4ca0-ac95-b001372a6b45" (UID: "87e6aee8-8be7-4ca0-ac95-b001372a6b45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:46:40.240753 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.240606    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-model-cache" (OuterVolumeSpecName: "model-cache") pod "87e6aee8-8be7-4ca0-ac95-b001372a6b45" (UID: "87e6aee8-8be7-4ca0-ac95-b001372a6b45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:46:40.240753 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.240721    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-tmp-dir\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:40.240753 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.240741    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-model-cache\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:40.240987 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.240966    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-home" (OuterVolumeSpecName: "home") pod "87e6aee8-8be7-4ca0-ac95-b001372a6b45" (UID: "87e6aee8-8be7-4ca0-ac95-b001372a6b45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:46:40.242621 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.242598    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e6aee8-8be7-4ca0-ac95-b001372a6b45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87e6aee8-8be7-4ca0-ac95-b001372a6b45" (UID: "87e6aee8-8be7-4ca0-ac95-b001372a6b45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:46:40.242715 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.242648    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-dshm" (OuterVolumeSpecName: "dshm") pod "87e6aee8-8be7-4ca0-ac95-b001372a6b45" (UID: "87e6aee8-8be7-4ca0-ac95-b001372a6b45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:46:40.242765 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.242707    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e6aee8-8be7-4ca0-ac95-b001372a6b45-kube-api-access-r6bc8" (OuterVolumeSpecName: "kube-api-access-r6bc8") pod "87e6aee8-8be7-4ca0-ac95-b001372a6b45" (UID: "87e6aee8-8be7-4ca0-ac95-b001372a6b45"). InnerVolumeSpecName "kube-api-access-r6bc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:46:40.317920 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.317888    2561 generic.go:356] "Generic (PLEG): container finished" podID="87e6aee8-8be7-4ca0-ac95-b001372a6b45" containerID="95b9d1d46abd24d5d7780c62777c047f011cb76786758d56b308bac2b79c0b18" exitCode=0
Sep 08 11:46:40.318089 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.317931    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"
Sep 08 11:46:40.318089 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.317945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs" event={"ID":"87e6aee8-8be7-4ca0-ac95-b001372a6b45","Type":"ContainerDied","Data":"95b9d1d46abd24d5d7780c62777c047f011cb76786758d56b308bac2b79c0b18"}
Sep 08 11:46:40.318089 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.317976    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs" event={"ID":"87e6aee8-8be7-4ca0-ac95-b001372a6b45","Type":"ContainerDied","Data":"648163665d345e4fb6a1dd9085b712d7acc524f7012fed5a3a39f589416526dc"}
Sep 08 11:46:40.318089 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.317991    2561 scope.go:117] "RemoveContainer" containerID="95b9d1d46abd24d5d7780c62777c047f011cb76786758d56b308bac2b79c0b18"
Sep 08 11:46:40.327187 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.327168    2561 scope.go:117] "RemoveContainer" containerID="95b9d1d46abd24d5d7780c62777c047f011cb76786758d56b308bac2b79c0b18"
Sep 08 11:46:40.327427 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:46:40.327408    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b9d1d46abd24d5d7780c62777c047f011cb76786758d56b308bac2b79c0b18\": container with ID starting with 95b9d1d46abd24d5d7780c62777c047f011cb76786758d56b308bac2b79c0b18 not found: ID does not exist" containerID="95b9d1d46abd24d5d7780c62777c047f011cb76786758d56b308bac2b79c0b18"
Sep 08 11:46:40.327583 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.327435    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b9d1d46abd24d5d7780c62777c047f011cb76786758d56b308bac2b79c0b18"} err="failed to get container status \"95b9d1d46abd24d5d7780c62777c047f011cb76786758d56b308bac2b79c0b18\": rpc error: code = NotFound desc = could not find container \"95b9d1d46abd24d5d7780c62777c047f011cb76786758d56b308bac2b79c0b18\": container with ID starting with 95b9d1d46abd24d5d7780c62777c047f011cb76786758d56b308bac2b79c0b18 not found: ID does not exist"
Sep 08 11:46:40.341369 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.341348    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87e6aee8-8be7-4ca0-ac95-b001372a6b45-tls-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:40.341369 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.341370    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6bc8\" (UniqueName: \"kubernetes.io/projected/87e6aee8-8be7-4ca0-ac95-b001372a6b45-kube-api-access-r6bc8\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:40.341502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.341379    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-home\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:40.341502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.341387    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87e6aee8-8be7-4ca0-ac95-b001372a6b45-dshm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:40.347108 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.347086    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"]
Sep 08 11:46:40.353143 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:40.353124    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-95669b/test-model-95669b-kserve-5f69646f6c-qsjfs"]
Sep 08 11:46:41.344562 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.344523    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7" podUID="abdd1645-2e04-4f98-8009-1e57b3c2c5b7" containerName="main" containerID="cri-o://3db413e39d92ae14631caf4a6d2b63a81959b6eb98fe1809cf916b8b16c0163f" gracePeriod=45
Sep 08 11:46:41.579920 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.579892    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:46:41.752876 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.752837    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-tmp-dir\") pod \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") "
Sep 08 11:46:41.753059 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.752911    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-model-cache\") pod \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") "
Sep 08 11:46:41.753059 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.752940    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-home\") pod \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") "
Sep 08 11:46:41.753059 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.752969    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-dshm\") pod \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") "
Sep 08 11:46:41.753059 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.753023    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22snp\" (UniqueName: \"kubernetes.io/projected/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-kube-api-access-22snp\") pod \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") "
Sep 08 11:46:41.753246 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.753057    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-tls-certs\") pod \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\" (UID: \"abdd1645-2e04-4f98-8009-1e57b3c2c5b7\") "
Sep 08 11:46:41.753246 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.753134    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "abdd1645-2e04-4f98-8009-1e57b3c2c5b7" (UID: "abdd1645-2e04-4f98-8009-1e57b3c2c5b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:46:41.753246 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.753185    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "abdd1645-2e04-4f98-8009-1e57b3c2c5b7" (UID: "abdd1645-2e04-4f98-8009-1e57b3c2c5b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:46:41.753352 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.753275    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-home" (OuterVolumeSpecName: "home") pod "abdd1645-2e04-4f98-8009-1e57b3c2c5b7" (UID: "abdd1645-2e04-4f98-8009-1e57b3c2c5b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:46:41.753352 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.753329    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-tmp-dir\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:41.753352 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.753348    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-model-cache\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:41.755330 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.755300    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-dshm" (OuterVolumeSpecName: "dshm") pod "abdd1645-2e04-4f98-8009-1e57b3c2c5b7" (UID: "abdd1645-2e04-4f98-8009-1e57b3c2c5b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:46:41.755440 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.755329    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-kube-api-access-22snp" (OuterVolumeSpecName: "kube-api-access-22snp") pod "abdd1645-2e04-4f98-8009-1e57b3c2c5b7" (UID: "abdd1645-2e04-4f98-8009-1e57b3c2c5b7"). InnerVolumeSpecName "kube-api-access-22snp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:46:41.755440 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.755350    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "abdd1645-2e04-4f98-8009-1e57b3c2c5b7" (UID: "abdd1645-2e04-4f98-8009-1e57b3c2c5b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:46:41.854097 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.854060    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-tls-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:41.854097 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.854088    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-home\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:41.854097 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.854097    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-dshm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:41.854097 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.854105    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22snp\" (UniqueName: \"kubernetes.io/projected/abdd1645-2e04-4f98-8009-1e57b3c2c5b7-kube-api-access-22snp\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:41.960226 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:41.960192    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87e6aee8-8be7-4ca0-ac95-b001372a6b45" path="/var/lib/kubelet/pods/87e6aee8-8be7-4ca0-ac95-b001372a6b45/volumes"
Sep 08 11:46:42.330536 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:42.330500    2561 generic.go:356] "Generic (PLEG): container finished" podID="abdd1645-2e04-4f98-8009-1e57b3c2c5b7" containerID="3db413e39d92ae14631caf4a6d2b63a81959b6eb98fe1809cf916b8b16c0163f" exitCode=0
Sep 08 11:46:42.330701 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:42.330562    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"
Sep 08 11:46:42.330701 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:42.330560    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7" event={"ID":"abdd1645-2e04-4f98-8009-1e57b3c2c5b7","Type":"ContainerDied","Data":"3db413e39d92ae14631caf4a6d2b63a81959b6eb98fe1809cf916b8b16c0163f"}
Sep 08 11:46:42.330701 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:42.330670    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7" event={"ID":"abdd1645-2e04-4f98-8009-1e57b3c2c5b7","Type":"ContainerDied","Data":"bd0d9c6ef9f3f3e36ee105594dc84e939bc2b68f2255a921c521c7cf01fc7cca"}
Sep 08 11:46:42.330701 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:42.330690    2561 scope.go:117] "RemoveContainer" containerID="3db413e39d92ae14631caf4a6d2b63a81959b6eb98fe1809cf916b8b16c0163f"
Sep 08 11:46:42.341144 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:42.341129    2561 scope.go:117] "RemoveContainer" containerID="3db413e39d92ae14631caf4a6d2b63a81959b6eb98fe1809cf916b8b16c0163f"
Sep 08 11:46:42.341635 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:46:42.341609    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db413e39d92ae14631caf4a6d2b63a81959b6eb98fe1809cf916b8b16c0163f\": container with ID starting with 3db413e39d92ae14631caf4a6d2b63a81959b6eb98fe1809cf916b8b16c0163f not found: ID does not exist" containerID="3db413e39d92ae14631caf4a6d2b63a81959b6eb98fe1809cf916b8b16c0163f"
Sep 08 11:46:42.341745 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:42.341640    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db413e39d92ae14631caf4a6d2b63a81959b6eb98fe1809cf916b8b16c0163f"} err="failed to get container status \"3db413e39d92ae14631caf4a6d2b63a81959b6eb98fe1809cf916b8b16c0163f\": rpc error: code = NotFound desc = could not find container \"3db413e39d92ae14631caf4a6d2b63a81959b6eb98fe1809cf916b8b16c0163f\": container with ID starting with 3db413e39d92ae14631caf4a6d2b63a81959b6eb98fe1809cf916b8b16c0163f not found: ID does not exist"
Sep 08 11:46:42.347970 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:42.347948    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"]
Sep 08 11:46:42.352764 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:42.352741    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-425b2c/test-model-425b2c-kserve-647c98d649-bzmk7"]
Sep 08 11:46:43.961056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:43.961018    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abdd1645-2e04-4f98-8009-1e57b3c2c5b7" path="/var/lib/kubelet/pods/abdd1645-2e04-4f98-8009-1e57b3c2c5b7/volumes"
Sep 08 11:46:44.728512 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:44.728459    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:45.313911 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.313874    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5"]
Sep 08 11:46:45.314363 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314343    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ff4118c-3392-42da-b846-7a7b2b471cc0" containerName="revoke-keys"
Sep 08 11:46:45.314407 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314369    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ff4118c-3392-42da-b846-7a7b2b471cc0" containerName="revoke-keys"
Sep 08 11:46:45.314407 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314393    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abdd1645-2e04-4f98-8009-1e57b3c2c5b7" containerName="main"
Sep 08 11:46:45.314407 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314402    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="abdd1645-2e04-4f98-8009-1e57b3c2c5b7" containerName="main"
Sep 08 11:46:45.314530 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314411    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e6aee8-8be7-4ca0-ac95-b001372a6b45" containerName="main"
Sep 08 11:46:45.314530 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314419    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e6aee8-8be7-4ca0-ac95-b001372a6b45" containerName="main"
Sep 08 11:46:45.314530 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314437    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d93e1465-3497-40b4-a67f-7232c883c545" containerName="payload-pre-processing"
Sep 08 11:46:45.314530 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314445    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d93e1465-3497-40b4-a67f-7232c883c545" containerName="payload-pre-processing"
Sep 08 11:46:45.314530 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314480    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a631a187-cc9d-4e74-ab8c-028b6013ff02" containerName="payload-processing"
Sep 08 11:46:45.314530 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314489    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a631a187-cc9d-4e74-ab8c-028b6013ff02" containerName="payload-processing"
Sep 08 11:46:45.314705 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314561    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ff4118c-3392-42da-b846-7a7b2b471cc0" containerName="revoke-keys"
Sep 08 11:46:45.314705 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314572    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d93e1465-3497-40b4-a67f-7232c883c545" containerName="payload-pre-processing"
Sep 08 11:46:45.314705 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314580    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="abdd1645-2e04-4f98-8009-1e57b3c2c5b7" containerName="main"
Sep 08 11:46:45.314705 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314590    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e6aee8-8be7-4ca0-ac95-b001372a6b45" containerName="main"
Sep 08 11:46:45.314705 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.314599    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a631a187-cc9d-4e74-ab8c-028b6013ff02" containerName="payload-processing"
Sep 08 11:46:45.319307 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.319288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5"
Sep 08 11:46:45.322234 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.322215    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:46:45.322403 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.322385    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:46:45.326534 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.326510    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5"]
Sep 08 11:46:45.384063 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.384030    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45dd2a1-2681-4840-b414-deab398c7086-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5\" (UID: \"d45dd2a1-2681-4840-b414-deab398c7086\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5"
Sep 08 11:46:45.485309 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.485269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45dd2a1-2681-4840-b414-deab398c7086-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5\" (UID: \"d45dd2a1-2681-4840-b414-deab398c7086\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5"
Sep 08 11:46:45.485968 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.485943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45dd2a1-2681-4840-b414-deab398c7086-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5\" (UID: \"d45dd2a1-2681-4840-b414-deab398c7086\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5"
Sep 08 11:46:45.631151 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.631067    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5"
Sep 08 11:46:45.904794 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:45.904761    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5"]
Sep 08 11:46:45.907004 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:46:45.906978    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45dd2a1_2681_4840_b414_deab398c7086.slice/crio-d7c8a5b7fc2f5ec9a1cddc6987266bde0e5c53429c8d60cec6b52340f7770820 WatchSource:0}: Error finding container d7c8a5b7fc2f5ec9a1cddc6987266bde0e5c53429c8d60cec6b52340f7770820: Status 404 returned error can't find the container with id d7c8a5b7fc2f5ec9a1cddc6987266bde0e5c53429c8d60cec6b52340f7770820
Sep 08 11:46:46.355864 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:46.355818    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5" event={"ID":"d45dd2a1-2681-4840-b414-deab398c7086","Type":"ContainerStarted","Data":"6790bdc7b6bbcca8d4c783f14d71d479ee417ca1143fff3e8f4947066e8b5134"}
Sep 08 11:46:46.356357 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:46.355935    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5" event={"ID":"d45dd2a1-2681-4840-b414-deab398c7086","Type":"ContainerStarted","Data":"d7c8a5b7fc2f5ec9a1cddc6987266bde0e5c53429c8d60cec6b52340f7770820"}
Sep 08 11:46:46.376798 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:46.376752    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5" podStartSLOduration=1.3767378049999999 podStartE2EDuration="1.376737805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:46:46.375218457 +0000 UTC m=+1012.941978997" watchObservedRunningTime="2026-09-08 11:46:46.376737805 +0000 UTC m=+1012.943498345"
Sep 08 11:46:46.797235 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:46.797197    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:47.362371 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:47.362335    2561 generic.go:356] "Generic (PLEG): container finished" podID="d45dd2a1-2681-4840-b414-deab398c7086" containerID="6790bdc7b6bbcca8d4c783f14d71d479ee417ca1143fff3e8f4947066e8b5134" exitCode=0
Sep 08 11:46:47.362766 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:47.362392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5" event={"ID":"d45dd2a1-2681-4840-b414-deab398c7086","Type":"ContainerDied","Data":"6790bdc7b6bbcca8d4c783f14d71d479ee417ca1143fff3e8f4947066e8b5134"}
Sep 08 11:46:48.494665 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:48.494639    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5"
Sep 08 11:46:48.508495 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:48.508462    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45dd2a1-2681-4840-b414-deab398c7086-maas-api-service-ca\") pod \"d45dd2a1-2681-4840-b414-deab398c7086\" (UID: \"d45dd2a1-2681-4840-b414-deab398c7086\") "
Sep 08 11:46:48.508812 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:48.508791    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45dd2a1-2681-4840-b414-deab398c7086-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d45dd2a1-2681-4840-b414-deab398c7086" (UID: "d45dd2a1-2681-4840-b414-deab398c7086"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:46:48.609514 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:48.609445    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45dd2a1-2681-4840-b414-deab398c7086-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:49.373992 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:49.373959    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5"
Sep 08 11:46:49.374195 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:49.373984    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5" event={"ID":"d45dd2a1-2681-4840-b414-deab398c7086","Type":"ContainerDied","Data":"d7c8a5b7fc2f5ec9a1cddc6987266bde0e5c53429c8d60cec6b52340f7770820"}
Sep 08 11:46:49.374195 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:49.374044    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7c8a5b7fc2f5ec9a1cddc6987266bde0e5c53429c8d60cec6b52340f7770820"
Sep 08 11:46:50.730534 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:50.730497    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:51.499508 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:51.499461    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:55.853388 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:55.853357    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-425b2c-29814465-k29ht"]
Sep 08 11:46:55.856618 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:55.856592    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-425b2c-29814465-k29ht"]
Sep 08 11:46:55.883179 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:55.883142    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"]
Sep 08 11:46:55.883424 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:55.883392    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c" podUID="ec97bdfa-8926-4c5c-9924-d06d84b68cbe" containerName="payload-processing" containerID="cri-o://771152659b9a61d8318e3fa35ccb13161c6315615c0bf5ce90e275f91b500e9d" gracePeriod=30
Sep 08 11:46:55.895939 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:55.895910    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:55.961125 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:55.961091    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a75ee979-910d-4bb8-bf75-f3dba2416c46" path="/var/lib/kubelet/pods/a75ee979-910d-4bb8-bf75-f3dba2416c46/volumes"
Sep 08 11:46:56.066502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.066451    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5"]
Sep 08 11:46:56.070700 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.070666    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425b2c-8pgl5"]
Sep 08 11:46:56.080707 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.080672    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:56.135629 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.135607    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"
Sep 08 11:46:56.173136 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.173107    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec97bdfa-8926-4c5c-9924-d06d84b68cbe-plugins-config-volume\") pod \"ec97bdfa-8926-4c5c-9924-d06d84b68cbe\" (UID: \"ec97bdfa-8926-4c5c-9924-d06d84b68cbe\") "
Sep 08 11:46:56.173307 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.173214    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwsw7\" (UniqueName: \"kubernetes.io/projected/ec97bdfa-8926-4c5c-9924-d06d84b68cbe-kube-api-access-zwsw7\") pod \"ec97bdfa-8926-4c5c-9924-d06d84b68cbe\" (UID: \"ec97bdfa-8926-4c5c-9924-d06d84b68cbe\") "
Sep 08 11:46:56.173577 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.173555    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec97bdfa-8926-4c5c-9924-d06d84b68cbe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec97bdfa-8926-4c5c-9924-d06d84b68cbe" (UID: "ec97bdfa-8926-4c5c-9924-d06d84b68cbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:46:56.175224 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.175203    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec97bdfa-8926-4c5c-9924-d06d84b68cbe-kube-api-access-zwsw7" (OuterVolumeSpecName: "kube-api-access-zwsw7") pod "ec97bdfa-8926-4c5c-9924-d06d84b68cbe" (UID: "ec97bdfa-8926-4c5c-9924-d06d84b68cbe"). InnerVolumeSpecName "kube-api-access-zwsw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:46:56.203359 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.203333    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:56.274392 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.274359    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwsw7\" (UniqueName: \"kubernetes.io/projected/ec97bdfa-8926-4c5c-9924-d06d84b68cbe-kube-api-access-zwsw7\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:56.274392 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.274388    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec97bdfa-8926-4c5c-9924-d06d84b68cbe-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:46:56.410707 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.410617    2561 generic.go:356] "Generic (PLEG): container finished" podID="ec97bdfa-8926-4c5c-9924-d06d84b68cbe" containerID="771152659b9a61d8318e3fa35ccb13161c6315615c0bf5ce90e275f91b500e9d" exitCode=0
Sep 08 11:46:56.410707 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.410662    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"
Sep 08 11:46:56.410923 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.410702    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c" event={"ID":"ec97bdfa-8926-4c5c-9924-d06d84b68cbe","Type":"ContainerDied","Data":"771152659b9a61d8318e3fa35ccb13161c6315615c0bf5ce90e275f91b500e9d"}
Sep 08 11:46:56.410923 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.410757    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c" event={"ID":"ec97bdfa-8926-4c5c-9924-d06d84b68cbe","Type":"ContainerDied","Data":"0d1635dfb0c8942b59530508f3e16c22e78a0f46f88b132f9628644da78282ed"}
Sep 08 11:46:56.410923 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.410781    2561 scope.go:117] "RemoveContainer" containerID="771152659b9a61d8318e3fa35ccb13161c6315615c0bf5ce90e275f91b500e9d"
Sep 08 11:46:56.419261 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.419243    2561 scope.go:117] "RemoveContainer" containerID="771152659b9a61d8318e3fa35ccb13161c6315615c0bf5ce90e275f91b500e9d"
Sep 08 11:46:56.419549 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:46:56.419530    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771152659b9a61d8318e3fa35ccb13161c6315615c0bf5ce90e275f91b500e9d\": container with ID starting with 771152659b9a61d8318e3fa35ccb13161c6315615c0bf5ce90e275f91b500e9d not found: ID does not exist" containerID="771152659b9a61d8318e3fa35ccb13161c6315615c0bf5ce90e275f91b500e9d"
Sep 08 11:46:56.419626 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.419561    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771152659b9a61d8318e3fa35ccb13161c6315615c0bf5ce90e275f91b500e9d"} err="failed to get container status \"771152659b9a61d8318e3fa35ccb13161c6315615c0bf5ce90e275f91b500e9d\": rpc error: code = NotFound desc = could not find container \"771152659b9a61d8318e3fa35ccb13161c6315615c0bf5ce90e275f91b500e9d\": container with ID starting with 771152659b9a61d8318e3fa35ccb13161c6315615c0bf5ce90e275f91b500e9d not found: ID does not exist"
Sep 08 11:46:56.439279 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.439250    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"]
Sep 08 11:46:56.444411 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.444380    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-425b2c-798c74d9bc-p994c"]
Sep 08 11:46:56.469398 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:56.469365    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:57.960594 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:57.960559    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d45dd2a1-2681-4840-b414-deab398c7086" path="/var/lib/kubelet/pods/d45dd2a1-2681-4840-b414-deab398c7086/volumes"
Sep 08 11:46:57.960952 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:57.960893    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec97bdfa-8926-4c5c-9924-d06d84b68cbe" path="/var/lib/kubelet/pods/ec97bdfa-8926-4c5c-9924-d06d84b68cbe/volumes"
Sep 08 11:46:58.320243 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.320171    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"]
Sep 08 11:46:58.320478 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.320456    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec97bdfa-8926-4c5c-9924-d06d84b68cbe" containerName="payload-processing"
Sep 08 11:46:58.320525 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.320481    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec97bdfa-8926-4c5c-9924-d06d84b68cbe" containerName="payload-processing"
Sep 08 11:46:58.320525 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.320495    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d45dd2a1-2681-4840-b414-deab398c7086" containerName="revoke-keys"
Sep 08 11:46:58.320525 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.320501    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d45dd2a1-2681-4840-b414-deab398c7086" containerName="revoke-keys"
Sep 08 11:46:58.320615 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.320568    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec97bdfa-8926-4c5c-9924-d06d84b68cbe" containerName="payload-processing"
Sep 08 11:46:58.320615 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.320577    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d45dd2a1-2681-4840-b414-deab398c7086" containerName="revoke-keys"
Sep 08 11:46:58.324665 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.324646    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.327364 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.327328    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-536240\"/\"openshift-service-ca.crt\""
Sep 08 11:46:58.328644 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.328622    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-536240\"/\"noten-model-536240-kserve-self-signed-certs\""
Sep 08 11:46:58.328758 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.328742    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-536240\"/\"default-dockercfg-hmhlf\""
Sep 08 11:46:58.329061 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.329041    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-536240\"/\"kube-root-ca.crt\""
Sep 08 11:46:58.335357 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.335335    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"]
Sep 08 11:46:58.373421 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.373390    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:58.391423 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.391393    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-model-cache\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.391549 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.391425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-dshm\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.391549 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.391453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2998f1f3-dbb1-41b6-8096-c1440744dcc2-tls-certs\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.391549 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.391528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjf88\" (UniqueName: \"kubernetes.io/projected/2998f1f3-dbb1-41b6-8096-c1440744dcc2-kube-api-access-gjf88\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.391663 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.391556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-tmp-dir\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.391663 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.391595    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-home\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.492177 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.492141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjf88\" (UniqueName: \"kubernetes.io/projected/2998f1f3-dbb1-41b6-8096-c1440744dcc2-kube-api-access-gjf88\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.492177 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.492180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-tmp-dir\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.492424 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.492202    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-home\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.492424 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.492238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-model-cache\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.492424 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.492256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-dshm\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.492424 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.492280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2998f1f3-dbb1-41b6-8096-c1440744dcc2-tls-certs\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.492671 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.492572    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-tmp-dir\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.492671 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.492622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-model-cache\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.492761 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.492685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-home\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.494723 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.494700    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-dshm\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.494909 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.494893    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2998f1f3-dbb1-41b6-8096-c1440744dcc2-tls-certs\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.501086 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.501065    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjf88\" (UniqueName: \"kubernetes.io/projected/2998f1f3-dbb1-41b6-8096-c1440744dcc2-kube-api-access-gjf88\") pod \"noten-model-536240-kserve-9c489df8c-bsdvs\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") " pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.634417 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.634333    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:46:58.867297 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.867261    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:58.906308 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.906287    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"]
Sep 08 11:46:58.908445 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:46:58.908420    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2998f1f3_dbb1_41b6_8096_c1440744dcc2.slice/crio-4ecd0d13407bb3840f3067499c0ea496c19cfc8aebdf200390e9bdf0791ede4c WatchSource:0}: Error finding container 4ecd0d13407bb3840f3067499c0ea496c19cfc8aebdf200390e9bdf0791ede4c: Status 404 returned error can't find the container with id 4ecd0d13407bb3840f3067499c0ea496c19cfc8aebdf200390e9bdf0791ede4c
Sep 08 11:46:58.976169 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:58.976141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:59.076647 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:59.076610    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:59.187530 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:59.187428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:59.426803 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:59.426767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs" event={"ID":"2998f1f3-dbb1-41b6-8096-c1440744dcc2","Type":"ContainerStarted","Data":"bd0d481c10ef400f11ddcf0971d2cf7b608d241c1a5da3566d5010723c094fbf"}
Sep 08 11:46:59.426983 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:59.426813    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs" event={"ID":"2998f1f3-dbb1-41b6-8096-c1440744dcc2","Type":"ContainerStarted","Data":"4ecd0d13407bb3840f3067499c0ea496c19cfc8aebdf200390e9bdf0791ede4c"}
Sep 08 11:46:59.452640 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:59.452519    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs" podStartSLOduration=1.452501148 podStartE2EDuration="1.452501148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:46:59.451665551 +0000 UTC m=+1026.018426102" watchObservedRunningTime="2026-09-08 11:46:59.452501148 +0000 UTC m=+1026.019261688"
Sep 08 11:46:59.787119 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:59.787080    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:59.889219 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:59.889191    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:46:59.986300 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:46:59.986249    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:01.455114 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:01.455082    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:03.511148 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:03.511112    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:03.611769 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:03.611709    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:03.768488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:03.768396    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:04.301048 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:04.301020    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:04.608973 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:04.608869    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:04.744311 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:04.744282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:07.996060 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:07.996025    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:08.112925 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:08.112889    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:08.634911 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:08.634870    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:47:08.635062 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:08.634929    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:47:08.647243 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:08.647218    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:47:08.651131 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:08.651103    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:08.735801 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:08.735779    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:09.182347 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.182316    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:09.281937 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.281905    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:09.382270 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.382232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:09.487590 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.487563    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:47:09.498410 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.498383    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:09.857497 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.857397    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"]
Sep 08 11:47:09.860455 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.860439    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"
Sep 08 11:47:09.862951 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.862926    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"]
Sep 08 11:47:09.863556 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.863539    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-838a68\""
Sep 08 11:47:09.866031 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.866012    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"
Sep 08 11:47:09.878284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.878260    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"]
Sep 08 11:47:09.881223 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.881197    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"]
Sep 08 11:47:09.987944 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.987915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62dfa631-bbc0-4811-a0ba-26b7e8b77638-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b\" (UID: \"62dfa631-bbc0-4811-a0ba-26b7e8b77638\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"
Sep 08 11:47:09.987944 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.987944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb4cr\" (UniqueName: \"kubernetes.io/projected/62dfa631-bbc0-4811-a0ba-26b7e8b77638-kube-api-access-cb4cr\") pod \"payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b\" (UID: \"62dfa631-bbc0-4811-a0ba-26b7e8b77638\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"
Sep 08 11:47:09.988142 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.988027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpkz5\" (UniqueName: \"kubernetes.io/projected/3cc81383-082f-47f2-90bd-72076fa5bfb3-kube-api-access-xpkz5\") pod \"payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt\" (UID: \"3cc81383-082f-47f2-90bd-72076fa5bfb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"
Sep 08 11:47:09.988142 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:09.988046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc81383-082f-47f2-90bd-72076fa5bfb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt\" (UID: \"3cc81383-082f-47f2-90bd-72076fa5bfb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"
Sep 08 11:47:10.089315 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.089284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62dfa631-bbc0-4811-a0ba-26b7e8b77638-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b\" (UID: \"62dfa631-bbc0-4811-a0ba-26b7e8b77638\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"
Sep 08 11:47:10.089549 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.089317    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb4cr\" (UniqueName: \"kubernetes.io/projected/62dfa631-bbc0-4811-a0ba-26b7e8b77638-kube-api-access-cb4cr\") pod \"payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b\" (UID: \"62dfa631-bbc0-4811-a0ba-26b7e8b77638\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"
Sep 08 11:47:10.089549 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.089382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpkz5\" (UniqueName: \"kubernetes.io/projected/3cc81383-082f-47f2-90bd-72076fa5bfb3-kube-api-access-xpkz5\") pod \"payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt\" (UID: \"3cc81383-082f-47f2-90bd-72076fa5bfb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"
Sep 08 11:47:10.089549 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.089402    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc81383-082f-47f2-90bd-72076fa5bfb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt\" (UID: \"3cc81383-082f-47f2-90bd-72076fa5bfb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"
Sep 08 11:47:10.090122 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.090102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc81383-082f-47f2-90bd-72076fa5bfb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt\" (UID: \"3cc81383-082f-47f2-90bd-72076fa5bfb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"
Sep 08 11:47:10.090194 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.090128    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62dfa631-bbc0-4811-a0ba-26b7e8b77638-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b\" (UID: \"62dfa631-bbc0-4811-a0ba-26b7e8b77638\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"
Sep 08 11:47:10.099040 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.099008    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb4cr\" (UniqueName: \"kubernetes.io/projected/62dfa631-bbc0-4811-a0ba-26b7e8b77638-kube-api-access-cb4cr\") pod \"payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b\" (UID: \"62dfa631-bbc0-4811-a0ba-26b7e8b77638\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"
Sep 08 11:47:10.099232 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.099214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpkz5\" (UniqueName: \"kubernetes.io/projected/3cc81383-082f-47f2-90bd-72076fa5bfb3-kube-api-access-xpkz5\") pod \"payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt\" (UID: \"3cc81383-082f-47f2-90bd-72076fa5bfb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"
Sep 08 11:47:10.117055 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.117003    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:10.172825 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.172795    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"
Sep 08 11:47:10.177857 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.177839    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"
Sep 08 11:47:10.208853 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.208818    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:10.467255 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.467119    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"]
Sep 08 11:47:10.468944 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.468919    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"]
Sep 08 11:47:10.469972 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:47:10.469950    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62dfa631_bbc0_4811_a0ba_26b7e8b77638.slice/crio-965b79e99e45cb5e863df7063cb8346bb3ce42ccc46de786993267ec060272da WatchSource:0}: Error finding container 965b79e99e45cb5e863df7063cb8346bb3ce42ccc46de786993267ec060272da: Status 404 returned error can't find the container with id 965b79e99e45cb5e863df7063cb8346bb3ce42ccc46de786993267ec060272da
Sep 08 11:47:10.470723 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:47:10.470676    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc81383_082f_47f2_90bd_72076fa5bfb3.slice/crio-1b8bf339a7cd7241772c231daee8cb52e9177b98220d549e1355a0f3252c9624 WatchSource:0}: Error finding container 1b8bf339a7cd7241772c231daee8cb52e9177b98220d549e1355a0f3252c9624: Status 404 returned error can't find the container with id 1b8bf339a7cd7241772c231daee8cb52e9177b98220d549e1355a0f3252c9624
Sep 08 11:47:10.481549 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.481410    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b" event={"ID":"62dfa631-bbc0-4811-a0ba-26b7e8b77638","Type":"ContainerStarted","Data":"965b79e99e45cb5e863df7063cb8346bb3ce42ccc46de786993267ec060272da"}
Sep 08 11:47:10.482648 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:10.482628    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt" event={"ID":"3cc81383-082f-47f2-90bd-72076fa5bfb3","Type":"ContainerStarted","Data":"1b8bf339a7cd7241772c231daee8cb52e9177b98220d549e1355a0f3252c9624"}
Sep 08 11:47:11.489001 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:11.488970    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b" event={"ID":"62dfa631-bbc0-4811-a0ba-26b7e8b77638","Type":"ContainerStarted","Data":"c99c9be264c4f703889a4d0c5c14be0bb4c069a8ca70cc3f80037f7bfe578191"}
Sep 08 11:47:11.489559 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:11.489071    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"
Sep 08 11:47:11.490574 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:11.490548    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt" event={"ID":"3cc81383-082f-47f2-90bd-72076fa5bfb3","Type":"ContainerStarted","Data":"433716011e96738bbb96b082fd24ae963f63b769a324a9e3f4108febe1e68924"}
Sep 08 11:47:11.490687 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:11.490592    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"
Sep 08 11:47:11.512350 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:11.512310    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b" podStartSLOduration=2.512297234 podStartE2EDuration="2.512297234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:47:11.511677923 +0000 UTC m=+1038.078438464" watchObservedRunningTime="2026-09-08 11:47:11.512297234 +0000 UTC m=+1038.079057775"
Sep 08 11:47:11.534665 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:11.534629    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt" podStartSLOduration=2.534617186 podStartE2EDuration="2.534617186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:47:11.533283601 +0000 UTC m=+1038.100044141" watchObservedRunningTime="2026-09-08 11:47:11.534617186 +0000 UTC m=+1038.101377727"
Sep 08 11:47:16.108075 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:16.108037    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:16.181970 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:16.181937    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:16.359108 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:16.359003    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:16.657483 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:16.657376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:17.147346 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:17.147313    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:20.118929 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:20.118883    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:20.220203 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:20.220167    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:21.454182 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:21.454143    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:21.554171 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:21.554141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:21.726005 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:21.725979    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:22.081602 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.081523    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:22.184276 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.184229    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:22.185158 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.185134    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"]
Sep 08 11:47:22.188360 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.188339    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"
Sep 08 11:47:22.191484 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.191446    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2fe44c90-dockercfg-wp85h\""
Sep 08 11:47:22.191587 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.191514    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2fe44c90\""
Sep 08 11:47:22.199502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.199479    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"]
Sep 08 11:47:22.289378 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.289337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcbc706-013d-4aa5-ae3e-b8bca73f07b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9\" (UID: \"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"
Sep 08 11:47:22.289378 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.289383    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkhbl\" (UniqueName: \"kubernetes.io/projected/fdcbc706-013d-4aa5-ae3e-b8bca73f07b8-kube-api-access-zkhbl\") pod \"payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9\" (UID: \"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"
Sep 08 11:47:22.390603 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.390519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcbc706-013d-4aa5-ae3e-b8bca73f07b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9\" (UID: \"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"
Sep 08 11:47:22.390603 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.390579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkhbl\" (UniqueName: \"kubernetes.io/projected/fdcbc706-013d-4aa5-ae3e-b8bca73f07b8-kube-api-access-zkhbl\") pod \"payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9\" (UID: \"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"
Sep 08 11:47:22.391193 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.391172    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcbc706-013d-4aa5-ae3e-b8bca73f07b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9\" (UID: \"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"
Sep 08 11:47:22.403536 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.403502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkhbl\" (UniqueName: \"kubernetes.io/projected/fdcbc706-013d-4aa5-ae3e-b8bca73f07b8-kube-api-access-zkhbl\") pod \"payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9\" (UID: \"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"
Sep 08 11:47:22.499708 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.499669    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"
Sep 08 11:47:22.501230 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.501198    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"
Sep 08 11:47:22.501531 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.501512    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"
Sep 08 11:47:22.773456 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:22.773392    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"]
Sep 08 11:47:22.776149 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:47:22.776122    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdcbc706_013d_4aa5_ae3e_b8bca73f07b8.slice/crio-f7cf398cc6dd78844767b1e0c56e5d91394cd06f99e67091756f2fc2c640452d WatchSource:0}: Error finding container f7cf398cc6dd78844767b1e0c56e5d91394cd06f99e67091756f2fc2c640452d: Status 404 returned error can't find the container with id f7cf398cc6dd78844767b1e0c56e5d91394cd06f99e67091756f2fc2c640452d
Sep 08 11:47:23.554454 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:23.554419    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9" event={"ID":"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8","Type":"ContainerStarted","Data":"a3f425f95a2a14ede0a39dcb921bbc8da723a6540a31a7f399807260d69fb364"}
Sep 08 11:47:23.554940 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:23.554489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9" event={"ID":"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8","Type":"ContainerStarted","Data":"f7cf398cc6dd78844767b1e0c56e5d91394cd06f99e67091756f2fc2c640452d"}
Sep 08 11:47:23.554940 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:23.554525    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"
Sep 08 11:47:23.575694 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:23.575641    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9" podStartSLOduration=1.5756212170000001 podStartE2EDuration="1.575621217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:47:23.572689173 +0000 UTC m=+1050.139449714" watchObservedRunningTime="2026-09-08 11:47:23.575621217 +0000 UTC m=+1050.142381763"
Sep 08 11:47:30.998359 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:30.998319    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"]
Sep 08 11:47:31.019058 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:31.019028    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:31.137366 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:31.137332    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:31.363130 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:31.363041    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:31.478554 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:31.478525    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:31.834086 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:31.834048    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:32.931795 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:32.931765    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:33.339121 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:33.339087    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:33.560793 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:33.560762    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:33.691460 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:33.691375    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:34.561656 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:34.561625    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"
Sep 08 11:47:36.185680 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:36.185647    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:36.278515 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:36.278456    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:36.584124 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:36.584093    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:37.365884 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:37.365845    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:37.441975 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:37.441939    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:37.570422 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:37.570387    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:37.929046 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:37.929000    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:38.029043 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.029007    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:38.029954 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.029936    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"]
Sep 08 11:47:38.033330 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.033313    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"
Sep 08 11:47:38.036244 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.036224    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-bb9b64\""
Sep 08 11:47:38.036373 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.036232    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-bb9b64-dockercfg-lbpmh\""
Sep 08 11:47:38.045226 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.045203    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"]
Sep 08 11:47:38.128754 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.128718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7xjh\" (UniqueName: \"kubernetes.io/projected/34400d9b-db76-4172-ba9b-b2354f0fcfc2-kube-api-access-k7xjh\") pod \"payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx\" (UID: \"34400d9b-db76-4172-ba9b-b2354f0fcfc2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"
Sep 08 11:47:38.128951 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.128773    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34400d9b-db76-4172-ba9b-b2354f0fcfc2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx\" (UID: \"34400d9b-db76-4172-ba9b-b2354f0fcfc2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"
Sep 08 11:47:38.229416 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.229387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7xjh\" (UniqueName: \"kubernetes.io/projected/34400d9b-db76-4172-ba9b-b2354f0fcfc2-kube-api-access-k7xjh\") pod \"payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx\" (UID: \"34400d9b-db76-4172-ba9b-b2354f0fcfc2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"
Sep 08 11:47:38.229594 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.229428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34400d9b-db76-4172-ba9b-b2354f0fcfc2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx\" (UID: \"34400d9b-db76-4172-ba9b-b2354f0fcfc2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"
Sep 08 11:47:38.230132 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.230111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34400d9b-db76-4172-ba9b-b2354f0fcfc2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx\" (UID: \"34400d9b-db76-4172-ba9b-b2354f0fcfc2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"
Sep 08 11:47:38.238641 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.238609    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7xjh\" (UniqueName: \"kubernetes.io/projected/34400d9b-db76-4172-ba9b-b2354f0fcfc2-kube-api-access-k7xjh\") pod \"payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx\" (UID: \"34400d9b-db76-4172-ba9b-b2354f0fcfc2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"
Sep 08 11:47:38.343626 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.343586    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"
Sep 08 11:47:38.619882 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.619765    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"]
Sep 08 11:47:38.622732 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:47:38.622697    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34400d9b_db76_4172_ba9b_b2354f0fcfc2.slice/crio-615f257bd78bc5c133cb22213b845b8cc2382f0d1f7562a80f9fd685aa17fdbb WatchSource:0}: Error finding container 615f257bd78bc5c133cb22213b845b8cc2382f0d1f7562a80f9fd685aa17fdbb: Status 404 returned error can't find the container with id 615f257bd78bc5c133cb22213b845b8cc2382f0d1f7562a80f9fd685aa17fdbb
Sep 08 11:47:38.635044 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:38.635019    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx" event={"ID":"34400d9b-db76-4172-ba9b-b2354f0fcfc2","Type":"ContainerStarted","Data":"615f257bd78bc5c133cb22213b845b8cc2382f0d1f7562a80f9fd685aa17fdbb"}
Sep 08 11:47:39.642695 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:39.642655    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx" event={"ID":"34400d9b-db76-4172-ba9b-b2354f0fcfc2","Type":"ContainerStarted","Data":"a60a7dc80bda451ebec9d8f460416ac0628ac90c626d54e6ed2cc3de63412374"}
Sep 08 11:47:39.643088 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:39.642732    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"
Sep 08 11:47:39.662976 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:39.662921    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx" podStartSLOduration=1.6629077190000001 podStartE2EDuration="1.662907719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:47:39.661317204 +0000 UTC m=+1066.228077742" watchObservedRunningTime="2026-09-08 11:47:39.662907719 +0000 UTC m=+1066.229668259"
Sep 08 11:47:42.510187 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:42.510154    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:45.057352 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:45.057320    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:45.671550 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:45.671514    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:46.017393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.017352    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs" podUID="2998f1f3-dbb1-41b6-8096-c1440744dcc2" containerName="main" containerID="cri-o://bd0d481c10ef400f11ddcf0971d2cf7b608d241c1a5da3566d5010723c094fbf" gracePeriod=45
Sep 08 11:47:46.117378 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.117308    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:46.220913 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.220880    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:46.255723 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.255701    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:47:46.296172 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.296108    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-dshm\") pod \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") "
Sep 08 11:47:46.296172 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.296140    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjf88\" (UniqueName: \"kubernetes.io/projected/2998f1f3-dbb1-41b6-8096-c1440744dcc2-kube-api-access-gjf88\") pod \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") "
Sep 08 11:47:46.296172 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.296165    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2998f1f3-dbb1-41b6-8096-c1440744dcc2-tls-certs\") pod \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") "
Sep 08 11:47:46.296416 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.296199    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-home\") pod \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") "
Sep 08 11:47:46.296416 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.296227    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-model-cache\") pod \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") "
Sep 08 11:47:46.296416 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.296277    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-tmp-dir\") pod \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\" (UID: \"2998f1f3-dbb1-41b6-8096-c1440744dcc2\") "
Sep 08 11:47:46.296657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.296532    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-home" (OuterVolumeSpecName: "home") pod "2998f1f3-dbb1-41b6-8096-c1440744dcc2" (UID: "2998f1f3-dbb1-41b6-8096-c1440744dcc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:47:46.296657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.296547    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-model-cache" (OuterVolumeSpecName: "model-cache") pod "2998f1f3-dbb1-41b6-8096-c1440744dcc2" (UID: "2998f1f3-dbb1-41b6-8096-c1440744dcc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:47:46.296781 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.296672    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2998f1f3-dbb1-41b6-8096-c1440744dcc2" (UID: "2998f1f3-dbb1-41b6-8096-c1440744dcc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:47:46.298420 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.298391    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-dshm" (OuterVolumeSpecName: "dshm") pod "2998f1f3-dbb1-41b6-8096-c1440744dcc2" (UID: "2998f1f3-dbb1-41b6-8096-c1440744dcc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:47:46.298420 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.298405    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2998f1f3-dbb1-41b6-8096-c1440744dcc2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2998f1f3-dbb1-41b6-8096-c1440744dcc2" (UID: "2998f1f3-dbb1-41b6-8096-c1440744dcc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:47:46.298621 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.298405    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2998f1f3-dbb1-41b6-8096-c1440744dcc2-kube-api-access-gjf88" (OuterVolumeSpecName: "kube-api-access-gjf88") pod "2998f1f3-dbb1-41b6-8096-c1440744dcc2" (UID: "2998f1f3-dbb1-41b6-8096-c1440744dcc2"). InnerVolumeSpecName "kube-api-access-gjf88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:47:46.397519 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.397491    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-dshm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:47:46.397519 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.397515    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjf88\" (UniqueName: \"kubernetes.io/projected/2998f1f3-dbb1-41b6-8096-c1440744dcc2-kube-api-access-gjf88\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:47:46.397691 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.397527    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2998f1f3-dbb1-41b6-8096-c1440744dcc2-tls-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:47:46.397691 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.397535    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-home\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:47:46.397691 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.397543    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-model-cache\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:47:46.397691 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.397551    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2998f1f3-dbb1-41b6-8096-c1440744dcc2-tmp-dir\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:47:46.677280 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.677201    2561 generic.go:356] "Generic (PLEG): container finished" podID="2998f1f3-dbb1-41b6-8096-c1440744dcc2" containerID="bd0d481c10ef400f11ddcf0971d2cf7b608d241c1a5da3566d5010723c094fbf" exitCode=0
Sep 08 11:47:46.677280 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.677243    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"
Sep 08 11:47:46.677280 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.677269    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs" event={"ID":"2998f1f3-dbb1-41b6-8096-c1440744dcc2","Type":"ContainerDied","Data":"bd0d481c10ef400f11ddcf0971d2cf7b608d241c1a5da3566d5010723c094fbf"}
Sep 08 11:47:46.677524 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.677298    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs" event={"ID":"2998f1f3-dbb1-41b6-8096-c1440744dcc2","Type":"ContainerDied","Data":"4ecd0d13407bb3840f3067499c0ea496c19cfc8aebdf200390e9bdf0791ede4c"}
Sep 08 11:47:46.677524 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.677316    2561 scope.go:117] "RemoveContainer" containerID="bd0d481c10ef400f11ddcf0971d2cf7b608d241c1a5da3566d5010723c094fbf"
Sep 08 11:47:46.686335 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.686318    2561 scope.go:117] "RemoveContainer" containerID="bd0d481c10ef400f11ddcf0971d2cf7b608d241c1a5da3566d5010723c094fbf"
Sep 08 11:47:46.686605 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:47:46.686588    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0d481c10ef400f11ddcf0971d2cf7b608d241c1a5da3566d5010723c094fbf\": container with ID starting with bd0d481c10ef400f11ddcf0971d2cf7b608d241c1a5da3566d5010723c094fbf not found: ID does not exist" containerID="bd0d481c10ef400f11ddcf0971d2cf7b608d241c1a5da3566d5010723c094fbf"
Sep 08 11:47:46.686675 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.686613    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0d481c10ef400f11ddcf0971d2cf7b608d241c1a5da3566d5010723c094fbf"} err="failed to get container status \"bd0d481c10ef400f11ddcf0971d2cf7b608d241c1a5da3566d5010723c094fbf\": rpc error: code = NotFound desc = could not find container \"bd0d481c10ef400f11ddcf0971d2cf7b608d241c1a5da3566d5010723c094fbf\": container with ID starting with bd0d481c10ef400f11ddcf0971d2cf7b608d241c1a5da3566d5010723c094fbf not found: ID does not exist"
Sep 08 11:47:46.700168 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.700145    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"]
Sep 08 11:47:46.706292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:46.706273    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-536240/noten-model-536240-kserve-9c489df8c-bsdvs"]
Sep 08 11:47:47.960359 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:47.960326    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2998f1f3-dbb1-41b6-8096-c1440744dcc2" path="/var/lib/kubelet/pods/2998f1f3-dbb1-41b6-8096-c1440744dcc2/volumes"
Sep 08 11:47:50.649650 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:50.649618    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"
Sep 08 11:47:52.676998 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:52.676960    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:52.744375 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:52.744341    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:47:53.040387 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.040354    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm"]
Sep 08 11:47:53.040703 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.040690    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2998f1f3-dbb1-41b6-8096-c1440744dcc2" containerName="main"
Sep 08 11:47:53.040748 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.040704    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2998f1f3-dbb1-41b6-8096-c1440744dcc2" containerName="main"
Sep 08 11:47:53.040786 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.040772    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2998f1f3-dbb1-41b6-8096-c1440744dcc2" containerName="main"
Sep 08 11:47:53.043450 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.043434    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm"
Sep 08 11:47:53.046246 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.046218    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:47:53.046246 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.046240    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:47:53.053217 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.053189    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm"]
Sep 08 11:47:53.151245 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.151214    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbd7e34-2154-4338-b579-465593f5476c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm\" (UID: \"0dbd7e34-2154-4338-b579-465593f5476c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm"
Sep 08 11:47:53.252641 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.252601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbd7e34-2154-4338-b579-465593f5476c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm\" (UID: \"0dbd7e34-2154-4338-b579-465593f5476c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm"
Sep 08 11:47:53.253236 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.253213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbd7e34-2154-4338-b579-465593f5476c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm\" (UID: \"0dbd7e34-2154-4338-b579-465593f5476c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm"
Sep 08 11:47:53.353353 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.353256    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm"
Sep 08 11:47:53.628218 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.628193    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm"]
Sep 08 11:47:53.629989 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:47:53.629956    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbd7e34_2154_4338_b579_465593f5476c.slice/crio-dd4383bef6371fe0f78b98d428ae1ef488ca5b4ae04260ce35d87499a53d8c5d WatchSource:0}: Error finding container dd4383bef6371fe0f78b98d428ae1ef488ca5b4ae04260ce35d87499a53d8c5d: Status 404 returned error can't find the container with id dd4383bef6371fe0f78b98d428ae1ef488ca5b4ae04260ce35d87499a53d8c5d
Sep 08 11:47:53.730505 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:53.730450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm" event={"ID":"0dbd7e34-2154-4338-b579-465593f5476c","Type":"ContainerStarted","Data":"dd4383bef6371fe0f78b98d428ae1ef488ca5b4ae04260ce35d87499a53d8c5d"}
Sep 08 11:47:54.736713 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:54.736685    2561 generic.go:356] "Generic (PLEG): container finished" podID="0dbd7e34-2154-4338-b579-465593f5476c" containerID="f425ee29a67d3088bc9063de9f60378adafb304d530f0719780a69e85f70cf8d" exitCode=0
Sep 08 11:47:54.737086 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:54.736740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm" event={"ID":"0dbd7e34-2154-4338-b579-465593f5476c","Type":"ContainerDied","Data":"f425ee29a67d3088bc9063de9f60378adafb304d530f0719780a69e85f70cf8d"}
Sep 08 11:47:55.868564 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:55.868540    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm"
Sep 08 11:47:55.975391 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:55.975361    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbd7e34-2154-4338-b579-465593f5476c-maas-api-service-ca\") pod \"0dbd7e34-2154-4338-b579-465593f5476c\" (UID: \"0dbd7e34-2154-4338-b579-465593f5476c\") "
Sep 08 11:47:55.975711 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:55.975689    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbd7e34-2154-4338-b579-465593f5476c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dbd7e34-2154-4338-b579-465593f5476c" (UID: "0dbd7e34-2154-4338-b579-465593f5476c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:47:56.076910 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:56.076840    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dbd7e34-2154-4338-b579-465593f5476c-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:47:56.747660 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:56.747626    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm"
Sep 08 11:47:56.747826 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:56.747630    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm" event={"ID":"0dbd7e34-2154-4338-b579-465593f5476c","Type":"ContainerDied","Data":"dd4383bef6371fe0f78b98d428ae1ef488ca5b4ae04260ce35d87499a53d8c5d"}
Sep 08 11:47:56.747826 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:56.747752    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd4383bef6371fe0f78b98d428ae1ef488ca5b4ae04260ce35d87499a53d8c5d"
Sep 08 11:47:58.353549 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:47:58.353512    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:01.127536 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:01.127494    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:03.211096 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:03.211059    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:03.832882 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:03.832844    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:03.872051 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:03.872019    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"]
Sep 08 11:48:03.872285 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:03.872258    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9" podUID="fdcbc706-013d-4aa5-ae3e-b8bca73f07b8" containerName="payload-pre-processing" containerID="cri-o://a3f425f95a2a14ede0a39dcb921bbc8da723a6540a31a7f399807260d69fb364" gracePeriod=30
Sep 08 11:48:03.925150 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:03.925118    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:04.025147 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.025096    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:04.041697 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.041666    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm"]
Sep 08 11:48:04.045892 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.045864    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fe44c90-stvjm"]
Sep 08 11:48:04.120355 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.120330    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"
Sep 08 11:48:04.125066 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.125041    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:04.242115 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.242084    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkhbl\" (UniqueName: \"kubernetes.io/projected/fdcbc706-013d-4aa5-ae3e-b8bca73f07b8-kube-api-access-zkhbl\") pod \"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8\" (UID: \"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8\") "
Sep 08 11:48:04.242115 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.242120    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcbc706-013d-4aa5-ae3e-b8bca73f07b8-plugins-config-volume\") pod \"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8\" (UID: \"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8\") "
Sep 08 11:48:04.242681 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.242629    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdcbc706-013d-4aa5-ae3e-b8bca73f07b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdcbc706-013d-4aa5-ae3e-b8bca73f07b8" (UID: "fdcbc706-013d-4aa5-ae3e-b8bca73f07b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:48:04.244381 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.244350    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdcbc706-013d-4aa5-ae3e-b8bca73f07b8-kube-api-access-zkhbl" (OuterVolumeSpecName: "kube-api-access-zkhbl") pod "fdcbc706-013d-4aa5-ae3e-b8bca73f07b8" (UID: "fdcbc706-013d-4aa5-ae3e-b8bca73f07b8"). InnerVolumeSpecName "kube-api-access-zkhbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:48:04.343069 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.343035    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkhbl\" (UniqueName: \"kubernetes.io/projected/fdcbc706-013d-4aa5-ae3e-b8bca73f07b8-kube-api-access-zkhbl\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:48:04.343069 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.343063    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcbc706-013d-4aa5-ae3e-b8bca73f07b8-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:48:04.386440 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.386338    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:04.790448 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.790408    2561 generic.go:356] "Generic (PLEG): container finished" podID="fdcbc706-013d-4aa5-ae3e-b8bca73f07b8" containerID="a3f425f95a2a14ede0a39dcb921bbc8da723a6540a31a7f399807260d69fb364" exitCode=0
Sep 08 11:48:04.790673 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.790456    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"
Sep 08 11:48:04.790673 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.790498    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9" event={"ID":"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8","Type":"ContainerDied","Data":"a3f425f95a2a14ede0a39dcb921bbc8da723a6540a31a7f399807260d69fb364"}
Sep 08 11:48:04.790673 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.790557    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9" event={"ID":"fdcbc706-013d-4aa5-ae3e-b8bca73f07b8","Type":"ContainerDied","Data":"f7cf398cc6dd78844767b1e0c56e5d91394cd06f99e67091756f2fc2c640452d"}
Sep 08 11:48:04.790673 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.790585    2561 scope.go:117] "RemoveContainer" containerID="a3f425f95a2a14ede0a39dcb921bbc8da723a6540a31a7f399807260d69fb364"
Sep 08 11:48:04.799865 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.799848    2561 scope.go:117] "RemoveContainer" containerID="a3f425f95a2a14ede0a39dcb921bbc8da723a6540a31a7f399807260d69fb364"
Sep 08 11:48:04.800145 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:48:04.800125    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f425f95a2a14ede0a39dcb921bbc8da723a6540a31a7f399807260d69fb364\": container with ID starting with a3f425f95a2a14ede0a39dcb921bbc8da723a6540a31a7f399807260d69fb364 not found: ID does not exist" containerID="a3f425f95a2a14ede0a39dcb921bbc8da723a6540a31a7f399807260d69fb364"
Sep 08 11:48:04.800219 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.800154    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f425f95a2a14ede0a39dcb921bbc8da723a6540a31a7f399807260d69fb364"} err="failed to get container status \"a3f425f95a2a14ede0a39dcb921bbc8da723a6540a31a7f399807260d69fb364\": rpc error: code = NotFound desc = could not find container \"a3f425f95a2a14ede0a39dcb921bbc8da723a6540a31a7f399807260d69fb364\": container with ID starting with a3f425f95a2a14ede0a39dcb921bbc8da723a6540a31a7f399807260d69fb364 not found: ID does not exist"
Sep 08 11:48:04.813130 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.813101    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"]
Sep 08 11:48:04.819222 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:04.819195    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fe44c90-68fb58b6-w68z9"]
Sep 08 11:48:05.960891 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:05.960856    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dbd7e34-2154-4338-b579-465593f5476c" path="/var/lib/kubelet/pods/0dbd7e34-2154-4338-b579-465593f5476c/volumes"
Sep 08 11:48:05.961265 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:05.961185    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdcbc706-013d-4aa5-ae3e-b8bca73f07b8" path="/var/lib/kubelet/pods/fdcbc706-013d-4aa5-ae3e-b8bca73f07b8/volumes"
Sep 08 11:48:08.508393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.508358    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"]
Sep 08 11:48:08.508786 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.508690    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dbd7e34-2154-4338-b579-465593f5476c" containerName="revoke-keys"
Sep 08 11:48:08.508786 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.508704    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dbd7e34-2154-4338-b579-465593f5476c" containerName="revoke-keys"
Sep 08 11:48:08.508786 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.508719    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdcbc706-013d-4aa5-ae3e-b8bca73f07b8" containerName="payload-pre-processing"
Sep 08 11:48:08.508786 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.508724    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdcbc706-013d-4aa5-ae3e-b8bca73f07b8" containerName="payload-pre-processing"
Sep 08 11:48:08.508917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.508795    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdcbc706-013d-4aa5-ae3e-b8bca73f07b8" containerName="payload-pre-processing"
Sep 08 11:48:08.508917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.508804    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dbd7e34-2154-4338-b579-465593f5476c" containerName="revoke-keys"
Sep 08 11:48:08.512911 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.512896    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.516790 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.516763    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-838a68\"/\"kube-root-ca.crt\""
Sep 08 11:48:08.516790 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.516774    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-838a68\"/\"auth-test-model-838a68-kserve-self-signed-certs\""
Sep 08 11:48:08.516790 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.516763    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-838a68\"/\"default-dockercfg-8k58w\""
Sep 08 11:48:08.516995 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.516849    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-838a68\"/\"openshift-service-ca.crt\""
Sep 08 11:48:08.522655 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.522636    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"]
Sep 08 11:48:08.576861 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.576822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-model-cache\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.577013 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.576867    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pt8f\" (UniqueName: \"kubernetes.io/projected/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-kube-api-access-9pt8f\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.577013 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.576891    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-home\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.577013 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.576983    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-tmp-dir\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.577184 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.577038    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-dshm\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.577184 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.577084    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-tls-certs\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.633012 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.632985    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:08.678258 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.678235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-dshm\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.678359 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.678267    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-tls-certs\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.678359 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.678296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-model-cache\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.678359 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.678313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pt8f\" (UniqueName: \"kubernetes.io/projected/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-kube-api-access-9pt8f\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.678359 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.678330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-home\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.678585 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.678517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-tmp-dir\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.678745 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.678709    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-model-cache\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.678863 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.678824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-tmp-dir\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.678863 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.678848    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-home\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.680618 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.680593    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-dshm\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.680753 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.680737    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-tls-certs\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.688139 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.688121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pt8f\" (UniqueName: \"kubernetes.io/projected/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-kube-api-access-9pt8f\") pod \"auth-test-model-838a68-kserve-58bdcdf475-vsx8q\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") " pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:08.822147 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:08.822080    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:09.094282 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:09.094256    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"]
Sep 08 11:48:09.096924 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:48:09.096898    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7f2eefa_77ff_4e77_9d7c_5ff716bc6e37.slice/crio-1f2bd6d17d260a58381b5aa220661c6c7bd886c21852c977cc433877476639ab WatchSource:0}: Error finding container 1f2bd6d17d260a58381b5aa220661c6c7bd886c21852c977cc433877476639ab: Status 404 returned error can't find the container with id 1f2bd6d17d260a58381b5aa220661c6c7bd886c21852c977cc433877476639ab
Sep 08 11:48:09.816226 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:09.816190    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q" event={"ID":"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37","Type":"ContainerStarted","Data":"cecfa448fce639b532be8342d09f995552f4a833ed76bb862ae802909a6f4c90"}
Sep 08 11:48:09.816226 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:09.816229    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q" event={"ID":"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37","Type":"ContainerStarted","Data":"1f2bd6d17d260a58381b5aa220661c6c7bd886c21852c977cc433877476639ab"}
Sep 08 11:48:09.836818 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:09.836770    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q" podStartSLOduration=1.836730599 podStartE2EDuration="1.836730599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:48:09.83541821 +0000 UTC m=+1096.402178753" watchObservedRunningTime="2026-09-08 11:48:09.836730599 +0000 UTC m=+1096.403491140"
Sep 08 11:48:11.167068 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:11.167033    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:11.266191 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:11.266161    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:11.641031 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:11.640996    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:11.746515 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:11.746454    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:11.846641 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:11.846601    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:12.170624 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:12.170573    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:14.277584 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:14.277551    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:15.047016 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:15.046982    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:18.508825 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:18.508788    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:18.617502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:18.617457    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:18.722660 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:18.722627    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:18.822923 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:18.822843    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:18.822923 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:18.822915    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:18.835384 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:18.835361    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:18.842356 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:18.842335    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:18.877079 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:18.877058    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:48:25.238817 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:25.238779    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:25.309381 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:25.309352    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:26.121960 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:26.121929    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:26.427757 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:26.427673    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:26.824184 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:26.824147    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:29.561195 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:29.561165    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:29.691832 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:29.691801    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:30.735257 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:30.735225    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:30.830322 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:30.830282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:30.894227 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:30.894198    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:31.361111 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.361080    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:31.464010 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.463973    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:31.477683 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.477650    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"]
Sep 08 11:48:31.480933 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.480914    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"
Sep 08 11:48:31.484517 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.484500    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-69606d\""
Sep 08 11:48:31.491506 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.491481    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"]
Sep 08 11:48:31.494714 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.494698    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"
Sep 08 11:48:31.494805 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.494738    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"]
Sep 08 11:48:31.497994 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.497974    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-69606d-dockercfg-tjg6w\""
Sep 08 11:48:31.512048 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.512023    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"]
Sep 08 11:48:31.571513 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.571483    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82d8j\" (UniqueName: \"kubernetes.io/projected/11d11076-e0eb-42eb-b785-714ca7611eda-kube-api-access-82d8j\") pod \"payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g\" (UID: \"11d11076-e0eb-42eb-b785-714ca7611eda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"
Sep 08 11:48:31.571654 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.571528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcqqc\" (UniqueName: \"kubernetes.io/projected/0cd702a7-54b6-47be-93b9-9b2d13792e4e-kube-api-access-wcqqc\") pod \"payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk\" (UID: \"0cd702a7-54b6-47be-93b9-9b2d13792e4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"
Sep 08 11:48:31.571719 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.571649    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd702a7-54b6-47be-93b9-9b2d13792e4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk\" (UID: \"0cd702a7-54b6-47be-93b9-9b2d13792e4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"
Sep 08 11:48:31.571719 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.571687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d11076-e0eb-42eb-b785-714ca7611eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g\" (UID: \"11d11076-e0eb-42eb-b785-714ca7611eda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"
Sep 08 11:48:31.672764 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.672684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd702a7-54b6-47be-93b9-9b2d13792e4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk\" (UID: \"0cd702a7-54b6-47be-93b9-9b2d13792e4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"
Sep 08 11:48:31.672764 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.672733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d11076-e0eb-42eb-b785-714ca7611eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g\" (UID: \"11d11076-e0eb-42eb-b785-714ca7611eda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"
Sep 08 11:48:31.672952 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.672767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82d8j\" (UniqueName: \"kubernetes.io/projected/11d11076-e0eb-42eb-b785-714ca7611eda-kube-api-access-82d8j\") pod \"payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g\" (UID: \"11d11076-e0eb-42eb-b785-714ca7611eda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"
Sep 08 11:48:31.672952 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.672823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcqqc\" (UniqueName: \"kubernetes.io/projected/0cd702a7-54b6-47be-93b9-9b2d13792e4e-kube-api-access-wcqqc\") pod \"payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk\" (UID: \"0cd702a7-54b6-47be-93b9-9b2d13792e4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"
Sep 08 11:48:31.673489 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.673450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d11076-e0eb-42eb-b785-714ca7611eda-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g\" (UID: \"11d11076-e0eb-42eb-b785-714ca7611eda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"
Sep 08 11:48:31.673658 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.673636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd702a7-54b6-47be-93b9-9b2d13792e4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk\" (UID: \"0cd702a7-54b6-47be-93b9-9b2d13792e4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"
Sep 08 11:48:31.681798 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.681777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcqqc\" (UniqueName: \"kubernetes.io/projected/0cd702a7-54b6-47be-93b9-9b2d13792e4e-kube-api-access-wcqqc\") pod \"payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk\" (UID: \"0cd702a7-54b6-47be-93b9-9b2d13792e4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"
Sep 08 11:48:31.681905 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.681852    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82d8j\" (UniqueName: \"kubernetes.io/projected/11d11076-e0eb-42eb-b785-714ca7611eda-kube-api-access-82d8j\") pod \"payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g\" (UID: \"11d11076-e0eb-42eb-b785-714ca7611eda\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"
Sep 08 11:48:31.791140 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.791106    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"
Sep 08 11:48:31.804339 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:31.804316    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"
Sep 08 11:48:32.069039 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:32.069006    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"]
Sep 08 11:48:32.072018 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:48:32.071988    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d11076_e0eb_42eb_b785_714ca7611eda.slice/crio-c77c74632ab0366b7447c511e2512ae7d7176736177f2bbda80c145c1dab72fe WatchSource:0}: Error finding container c77c74632ab0366b7447c511e2512ae7d7176736177f2bbda80c145c1dab72fe: Status 404 returned error can't find the container with id c77c74632ab0366b7447c511e2512ae7d7176736177f2bbda80c145c1dab72fe
Sep 08 11:48:32.088890 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:32.088870    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"]
Sep 08 11:48:32.090196 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:48:32.090173    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd702a7_54b6_47be_93b9_9b2d13792e4e.slice/crio-1a00d98837bcd96cd3b026deed78fe6feb184fe5c65860be09f9a9aa1e4593e6 WatchSource:0}: Error finding container 1a00d98837bcd96cd3b026deed78fe6feb184fe5c65860be09f9a9aa1e4593e6: Status 404 returned error can't find the container with id 1a00d98837bcd96cd3b026deed78fe6feb184fe5c65860be09f9a9aa1e4593e6
Sep 08 11:48:32.938540 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:32.938505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g" event={"ID":"11d11076-e0eb-42eb-b785-714ca7611eda","Type":"ContainerStarted","Data":"1f311ee8dfc5911c2bfd747b914bf1c27a16b97499665ea0007ad6fed6145cef"}
Sep 08 11:48:32.938993 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:32.938571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g" event={"ID":"11d11076-e0eb-42eb-b785-714ca7611eda","Type":"ContainerStarted","Data":"c77c74632ab0366b7447c511e2512ae7d7176736177f2bbda80c145c1dab72fe"}
Sep 08 11:48:32.938993 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:32.938662    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"
Sep 08 11:48:32.940119 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:32.940094    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk" event={"ID":"0cd702a7-54b6-47be-93b9-9b2d13792e4e","Type":"ContainerStarted","Data":"1b62542f1a5d309dafab39d17660de85b49f1ae4b88edba15e2d593d57f7fdfb"}
Sep 08 11:48:32.940237 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:32.940140    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk" event={"ID":"0cd702a7-54b6-47be-93b9-9b2d13792e4e","Type":"ContainerStarted","Data":"1a00d98837bcd96cd3b026deed78fe6feb184fe5c65860be09f9a9aa1e4593e6"}
Sep 08 11:48:32.940237 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:32.940170    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"
Sep 08 11:48:32.956765 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:32.956719    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g" podStartSLOduration=1.956705286 podStartE2EDuration="1.956705286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:48:32.955936988 +0000 UTC m=+1119.522697530" watchObservedRunningTime="2026-09-08 11:48:32.956705286 +0000 UTC m=+1119.523465827"
Sep 08 11:48:32.975745 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:32.975681    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk" podStartSLOduration=1.975662248 podStartE2EDuration="1.975662248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:48:32.97412147 +0000 UTC m=+1119.540882013" watchObservedRunningTime="2026-09-08 11:48:32.975662248 +0000 UTC m=+1119.542422790"
Sep 08 11:48:33.020146 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:33.020117    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:43.946979 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:43.946946    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"
Sep 08 11:48:43.947418 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:43.947113    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"
Sep 08 11:48:54.501878 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:54.501849    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:54.841233 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:54.841109    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:55.164319 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:55.164238    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:55.563275 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:55.563240    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:58.388180 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:58.388121    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:58.478183 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:58.478157    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:59.614654 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:59.614618    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:48:59.712458 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:48:59.712427    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:00.129010 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.128975    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:00.231126 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.231089    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:00.336498 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.336448    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:00.348116 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.348088    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"]
Sep 08 11:49:00.352376 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.352361    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"
Sep 08 11:49:00.358660 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.358416    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-489cdd-dockercfg-9xsx2\""
Sep 08 11:49:00.358873 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.358854    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-489cdd\""
Sep 08 11:49:00.372016 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.371994    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"]
Sep 08 11:49:00.427958 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.427881    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62de157b-2c32-4c30-a40c-d8ffef499253-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5\" (UID: \"62de157b-2c32-4c30-a40c-d8ffef499253\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"
Sep 08 11:49:00.427958 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.427937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prfns\" (UniqueName: \"kubernetes.io/projected/62de157b-2c32-4c30-a40c-d8ffef499253-kube-api-access-prfns\") pod \"payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5\" (UID: \"62de157b-2c32-4c30-a40c-d8ffef499253\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"
Sep 08 11:49:00.529207 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.529171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prfns\" (UniqueName: \"kubernetes.io/projected/62de157b-2c32-4c30-a40c-d8ffef499253-kube-api-access-prfns\") pod \"payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5\" (UID: \"62de157b-2c32-4c30-a40c-d8ffef499253\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"
Sep 08 11:49:00.529373 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.529238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62de157b-2c32-4c30-a40c-d8ffef499253-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5\" (UID: \"62de157b-2c32-4c30-a40c-d8ffef499253\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"
Sep 08 11:49:00.529915 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.529895    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62de157b-2c32-4c30-a40c-d8ffef499253-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5\" (UID: \"62de157b-2c32-4c30-a40c-d8ffef499253\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"
Sep 08 11:49:00.539079 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.539055    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prfns\" (UniqueName: \"kubernetes.io/projected/62de157b-2c32-4c30-a40c-d8ffef499253-kube-api-access-prfns\") pod \"payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5\" (UID: \"62de157b-2c32-4c30-a40c-d8ffef499253\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"
Sep 08 11:49:00.662346 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.662323    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"
Sep 08 11:49:00.936831 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:00.936807    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"]
Sep 08 11:49:00.938942 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:49:00.938915    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62de157b_2c32_4c30_a40c_d8ffef499253.slice/crio-1c3cfd2ba88aeecf184750e898a6527b815c840fb958a0092ea5d901044fe8d6 WatchSource:0}: Error finding container 1c3cfd2ba88aeecf184750e898a6527b815c840fb958a0092ea5d901044fe8d6: Status 404 returned error can't find the container with id 1c3cfd2ba88aeecf184750e898a6527b815c840fb958a0092ea5d901044fe8d6
Sep 08 11:49:01.097082 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:01.097047    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5" event={"ID":"62de157b-2c32-4c30-a40c-d8ffef499253","Type":"ContainerStarted","Data":"2e2cd787661a927d1a536b3a26680d659ce788541ee68e05a1b13dcdb57fe303"}
Sep 08 11:49:01.097243 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:01.097123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5" event={"ID":"62de157b-2c32-4c30-a40c-d8ffef499253","Type":"ContainerStarted","Data":"1c3cfd2ba88aeecf184750e898a6527b815c840fb958a0092ea5d901044fe8d6"}
Sep 08 11:49:01.097243 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:01.097178    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"
Sep 08 11:49:01.117812 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:01.117756    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5" podStartSLOduration=1.117739962 podStartE2EDuration="1.117739962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:49:01.117378594 +0000 UTC m=+1147.684139136" watchObservedRunningTime="2026-09-08 11:49:01.117739962 +0000 UTC m=+1147.684500507"
Sep 08 11:49:12.105649 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:12.105615    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"
Sep 08 11:49:23.375917 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:23.375831    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:24.015192 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:24.015160    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7437-139668628981776"]
Sep 08 11:49:24.018308 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:24.018292    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:24.020996 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:24.020975    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zbjsb\""
Sep 08 11:49:24.026678 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:24.026654    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-139668628981776"]
Sep 08 11:49:24.133870 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:24.133831    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrj5r\" (UniqueName: \"kubernetes.io/projected/dab7487c-44ab-42e3-8d3e-552558e382a6-kube-api-access-zrj5r\") pod \"test-curl-7437-139668628981776\" (UID: \"dab7487c-44ab-42e3-8d3e-552558e382a6\") " pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:24.234415 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:24.234372    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrj5r\" (UniqueName: \"kubernetes.io/projected/dab7487c-44ab-42e3-8d3e-552558e382a6-kube-api-access-zrj5r\") pod \"test-curl-7437-139668628981776\" (UID: \"dab7487c-44ab-42e3-8d3e-552558e382a6\") " pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:24.243755 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:24.243720    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrj5r\" (UniqueName: \"kubernetes.io/projected/dab7487c-44ab-42e3-8d3e-552558e382a6-kube-api-access-zrj5r\") pod \"test-curl-7437-139668628981776\" (UID: \"dab7487c-44ab-42e3-8d3e-552558e382a6\") " pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:24.328721 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:24.328634    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:24.607757 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:24.607728    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-139668628981776"]
Sep 08 11:49:24.612939 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:49:24.612910    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddab7487c_44ab_42e3_8d3e_552558e382a6.slice/crio-9b12143fdd012a8d76fc558808516ee07242be0f9a94cecd8b377b8c811c8507 WatchSource:0}: Error finding container 9b12143fdd012a8d76fc558808516ee07242be0f9a94cecd8b377b8c811c8507: Status 404 returned error can't find the container with id 9b12143fdd012a8d76fc558808516ee07242be0f9a94cecd8b377b8c811c8507
Sep 08 11:49:24.614844 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:24.614825    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 11:49:25.217339 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:25.217303    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-139668628981776" event={"ID":"dab7487c-44ab-42e3-8d3e-552558e382a6","Type":"ContainerStarted","Data":"9b12143fdd012a8d76fc558808516ee07242be0f9a94cecd8b377b8c811c8507"}
Sep 08 11:49:26.224122 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:26.224087    2561 generic.go:356] "Generic (PLEG): container finished" podID="dab7487c-44ab-42e3-8d3e-552558e382a6" containerID="29050e5ab421b73d7db8cab4eb9c29121c45813e25b1b7e673d0d2854c15460e" exitCode=0
Sep 08 11:49:26.224495 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:26.224172    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-139668628981776" event={"ID":"dab7487c-44ab-42e3-8d3e-552558e382a6","Type":"ContainerDied","Data":"29050e5ab421b73d7db8cab4eb9c29121c45813e25b1b7e673d0d2854c15460e"}
Sep 08 11:49:27.360582 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.360556    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:27.383593 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.383565    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7437-139668628981776_dab7487c-44ab-42e3-8d3e-552558e382a6/test-curl-7437-139668628981776/0.log"
Sep 08 11:49:27.406475 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.406432    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7437-139668628981776"]
Sep 08 11:49:27.412831 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.412804    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7437-139668628981776"]
Sep 08 11:49:27.463326 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.463297    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrj5r\" (UniqueName: \"kubernetes.io/projected/dab7487c-44ab-42e3-8d3e-552558e382a6-kube-api-access-zrj5r\") pod \"dab7487c-44ab-42e3-8d3e-552558e382a6\" (UID: \"dab7487c-44ab-42e3-8d3e-552558e382a6\") "
Sep 08 11:49:27.465304 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.465274    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab7487c-44ab-42e3-8d3e-552558e382a6-kube-api-access-zrj5r" (OuterVolumeSpecName: "kube-api-access-zrj5r") pod "dab7487c-44ab-42e3-8d3e-552558e382a6" (UID: "dab7487c-44ab-42e3-8d3e-552558e382a6"). InnerVolumeSpecName "kube-api-access-zrj5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:49:27.535445 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.535371    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7437-139668628981776"]
Sep 08 11:49:27.535849 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.535829    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab7487c-44ab-42e3-8d3e-552558e382a6" containerName="test-curl-7437-139668628981776"
Sep 08 11:49:27.535939 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.535851    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab7487c-44ab-42e3-8d3e-552558e382a6" containerName="test-curl-7437-139668628981776"
Sep 08 11:49:27.535939 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.535930    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab7487c-44ab-42e3-8d3e-552558e382a6" containerName="test-curl-7437-139668628981776"
Sep 08 11:49:27.538870 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.538850    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:27.546033 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.546007    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-139668628981776"]
Sep 08 11:49:27.564028 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.564006    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrj5r\" (UniqueName: \"kubernetes.io/projected/dab7487c-44ab-42e3-8d3e-552558e382a6-kube-api-access-zrj5r\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:49:27.664689 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.664660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmq6s\" (UniqueName: \"kubernetes.io/projected/cd0920d8-9c17-474a-954f-afd81a790003-kube-api-access-fmq6s\") pod \"test-curl-7437-139668628981776\" (UID: \"cd0920d8-9c17-474a-954f-afd81a790003\") " pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:27.766084 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.766044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmq6s\" (UniqueName: \"kubernetes.io/projected/cd0920d8-9c17-474a-954f-afd81a790003-kube-api-access-fmq6s\") pod \"test-curl-7437-139668628981776\" (UID: \"cd0920d8-9c17-474a-954f-afd81a790003\") " pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:27.775197 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.775164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmq6s\" (UniqueName: \"kubernetes.io/projected/cd0920d8-9c17-474a-954f-afd81a790003-kube-api-access-fmq6s\") pod \"test-curl-7437-139668628981776\" (UID: \"cd0920d8-9c17-474a-954f-afd81a790003\") " pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:27.848455 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.848367    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:27.961360 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:27.961329    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab7487c-44ab-42e3-8d3e-552558e382a6" path="/var/lib/kubelet/pods/dab7487c-44ab-42e3-8d3e-552558e382a6/volumes"
Sep 08 11:49:28.120987 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:28.120916    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-139668628981776"]
Sep 08 11:49:28.123179 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:49:28.123156    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd0920d8_9c17_474a_954f_afd81a790003.slice/crio-c12f4e48ca37b13ded886bcdc300a6c07966918d7613c5068ddbb58b244fd63f WatchSource:0}: Error finding container c12f4e48ca37b13ded886bcdc300a6c07966918d7613c5068ddbb58b244fd63f: Status 404 returned error can't find the container with id c12f4e48ca37b13ded886bcdc300a6c07966918d7613c5068ddbb58b244fd63f
Sep 08 11:49:28.234871 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:28.234834    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-139668628981776" event={"ID":"cd0920d8-9c17-474a-954f-afd81a790003","Type":"ContainerStarted","Data":"c12f4e48ca37b13ded886bcdc300a6c07966918d7613c5068ddbb58b244fd63f"}
Sep 08 11:49:28.236071 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:28.236054    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:28.236177 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:28.236058    2561 scope.go:117] "RemoveContainer" containerID="29050e5ab421b73d7db8cab4eb9c29121c45813e25b1b7e673d0d2854c15460e"
Sep 08 11:49:29.244129 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:29.244097    2561 generic.go:356] "Generic (PLEG): container finished" podID="cd0920d8-9c17-474a-954f-afd81a790003" containerID="1b5595a576aafc844a2aaa4f91de972857781a7294aeff09c304d81d151d2a2c" exitCode=0
Sep 08 11:49:29.244583 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:29.244127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-139668628981776" event={"ID":"cd0920d8-9c17-474a-954f-afd81a790003","Type":"ContainerDied","Data":"1b5595a576aafc844a2aaa4f91de972857781a7294aeff09c304d81d151d2a2c"}
Sep 08 11:49:30.374263 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.374238    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:30.399270 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.399244    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7437-139668628981776_cd0920d8-9c17-474a-954f-afd81a790003/test-curl-7437-139668628981776/0.log"
Sep 08 11:49:30.427315 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.427280    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7437-139668628981776"]
Sep 08 11:49:30.431698 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.431675    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7437-139668628981776"]
Sep 08 11:49:30.489969 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.489943    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmq6s\" (UniqueName: \"kubernetes.io/projected/cd0920d8-9c17-474a-954f-afd81a790003-kube-api-access-fmq6s\") pod \"cd0920d8-9c17-474a-954f-afd81a790003\" (UID: \"cd0920d8-9c17-474a-954f-afd81a790003\") "
Sep 08 11:49:30.492089 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.492061    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0920d8-9c17-474a-954f-afd81a790003-kube-api-access-fmq6s" (OuterVolumeSpecName: "kube-api-access-fmq6s") pod "cd0920d8-9c17-474a-954f-afd81a790003" (UID: "cd0920d8-9c17-474a-954f-afd81a790003"). InnerVolumeSpecName "kube-api-access-fmq6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:49:30.550235 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.550159    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7437-139668628982352"]
Sep 08 11:49:30.550510 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.550495    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd0920d8-9c17-474a-954f-afd81a790003" containerName="test-curl-7437-139668628981776"
Sep 08 11:49:30.550557 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.550511    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd0920d8-9c17-474a-954f-afd81a790003" containerName="test-curl-7437-139668628981776"
Sep 08 11:49:30.550592 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.550576    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd0920d8-9c17-474a-954f-afd81a790003" containerName="test-curl-7437-139668628981776"
Sep 08 11:49:30.553294 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.553277    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:30.558802 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.558776    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-139668628982352"]
Sep 08 11:49:30.591368 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.591342    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmq6s\" (UniqueName: \"kubernetes.io/projected/cd0920d8-9c17-474a-954f-afd81a790003-kube-api-access-fmq6s\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:49:30.692000 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.691961    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh86d\" (UniqueName: \"kubernetes.io/projected/09463cbc-4c99-4a4c-9634-e48be0e680fb-kube-api-access-hh86d\") pod \"test-curl-7437-139668628982352\" (UID: \"09463cbc-4c99-4a4c-9634-e48be0e680fb\") " pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:30.792858 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.792826    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh86d\" (UniqueName: \"kubernetes.io/projected/09463cbc-4c99-4a4c-9634-e48be0e680fb-kube-api-access-hh86d\") pod \"test-curl-7437-139668628982352\" (UID: \"09463cbc-4c99-4a4c-9634-e48be0e680fb\") " pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:30.803305 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.803240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh86d\" (UniqueName: \"kubernetes.io/projected/09463cbc-4c99-4a4c-9634-e48be0e680fb-kube-api-access-hh86d\") pod \"test-curl-7437-139668628982352\" (UID: \"09463cbc-4c99-4a4c-9634-e48be0e680fb\") " pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:30.864118 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:30.864086    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:31.135432 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:31.135404    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-139668628982352"]
Sep 08 11:49:31.137592 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:49:31.137565    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09463cbc_4c99_4a4c_9634_e48be0e680fb.slice/crio-112e9f7138d79a29a6da57d47fbea49be6f23d57ebed32b33d2cdf8aa4c5cb54 WatchSource:0}: Error finding container 112e9f7138d79a29a6da57d47fbea49be6f23d57ebed32b33d2cdf8aa4c5cb54: Status 404 returned error can't find the container with id 112e9f7138d79a29a6da57d47fbea49be6f23d57ebed32b33d2cdf8aa4c5cb54
Sep 08 11:49:31.256761 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:31.256722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-139668628982352" event={"ID":"09463cbc-4c99-4a4c-9634-e48be0e680fb","Type":"ContainerStarted","Data":"112e9f7138d79a29a6da57d47fbea49be6f23d57ebed32b33d2cdf8aa4c5cb54"}
Sep 08 11:49:31.257856 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:31.257834    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c12f4e48ca37b13ded886bcdc300a6c07966918d7613c5068ddbb58b244fd63f"
Sep 08 11:49:31.257995 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:31.257887    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:31.268701 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:49:31.268673    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7437-139668628981776\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="cd0920d8-9c17-474a-954f-afd81a790003" pod="opendatahub/test-curl-7437-139668628981776"
Sep 08 11:49:31.961823 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:31.961789    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd0920d8-9c17-474a-954f-afd81a790003" path="/var/lib/kubelet/pods/cd0920d8-9c17-474a-954f-afd81a790003/volumes"
Sep 08 11:49:34.275618 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:34.275581    2561 generic.go:356] "Generic (PLEG): container finished" podID="09463cbc-4c99-4a4c-9634-e48be0e680fb" containerID="8bdf9e1999f52a442177ef34edd2390b4d4256788fcddcd0a1a78a3b114b0f7b" exitCode=0
Sep 08 11:49:34.275973 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:34.275662    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-139668628982352" event={"ID":"09463cbc-4c99-4a4c-9634-e48be0e680fb","Type":"ContainerDied","Data":"8bdf9e1999f52a442177ef34edd2390b4d4256788fcddcd0a1a78a3b114b0f7b"}
Sep 08 11:49:35.020607 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.020578    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:35.091311 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.091277    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:35.409908 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.409886    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:35.433197 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.433176    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7437-139668628982352_09463cbc-4c99-4a4c-9634-e48be0e680fb/test-curl-7437-139668628982352/0.log"
Sep 08 11:49:35.433328 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.433289    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh86d\" (UniqueName: \"kubernetes.io/projected/09463cbc-4c99-4a4c-9634-e48be0e680fb-kube-api-access-hh86d\") pod \"09463cbc-4c99-4a4c-9634-e48be0e680fb\" (UID: \"09463cbc-4c99-4a4c-9634-e48be0e680fb\") "
Sep 08 11:49:35.438509 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.436278    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09463cbc-4c99-4a4c-9634-e48be0e680fb-kube-api-access-hh86d" (OuterVolumeSpecName: "kube-api-access-hh86d") pod "09463cbc-4c99-4a4c-9634-e48be0e680fb" (UID: "09463cbc-4c99-4a4c-9634-e48be0e680fb"). InnerVolumeSpecName "kube-api-access-hh86d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:49:35.456293 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.456271    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7437-139668628982352"]
Sep 08 11:49:35.460988 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.460969    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7437-139668628982352"]
Sep 08 11:49:35.534772 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.534719    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh86d\" (UniqueName: \"kubernetes.io/projected/09463cbc-4c99-4a4c-9634-e48be0e680fb-kube-api-access-hh86d\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:49:35.583869 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.583843    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7437-139668628982352"]
Sep 08 11:49:35.584212 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.584199    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09463cbc-4c99-4a4c-9634-e48be0e680fb" containerName="test-curl-7437-139668628982352"
Sep 08 11:49:35.584262 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.584213    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="09463cbc-4c99-4a4c-9634-e48be0e680fb" containerName="test-curl-7437-139668628982352"
Sep 08 11:49:35.584299 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.584269    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="09463cbc-4c99-4a4c-9634-e48be0e680fb" containerName="test-curl-7437-139668628982352"
Sep 08 11:49:35.587178 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.587164    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:35.593400 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.593375    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-139668628982352"]
Sep 08 11:49:35.635063 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.635040    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcz5\" (UniqueName: \"kubernetes.io/projected/4a5c4769-19ad-415f-b268-8f5a352ba651-kube-api-access-mzcz5\") pod \"test-curl-7437-139668628982352\" (UID: \"4a5c4769-19ad-415f-b268-8f5a352ba651\") " pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:35.735686 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.735654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcz5\" (UniqueName: \"kubernetes.io/projected/4a5c4769-19ad-415f-b268-8f5a352ba651-kube-api-access-mzcz5\") pod \"test-curl-7437-139668628982352\" (UID: \"4a5c4769-19ad-415f-b268-8f5a352ba651\") " pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:35.747820 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.747794    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcz5\" (UniqueName: \"kubernetes.io/projected/4a5c4769-19ad-415f-b268-8f5a352ba651-kube-api-access-mzcz5\") pod \"test-curl-7437-139668628982352\" (UID: \"4a5c4769-19ad-415f-b268-8f5a352ba651\") " pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:35.896569 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.896499    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:35.962694 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:35.962667    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09463cbc-4c99-4a4c-9634-e48be0e680fb" path="/var/lib/kubelet/pods/09463cbc-4c99-4a4c-9634-e48be0e680fb/volumes"
Sep 08 11:49:36.168046 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:36.168022    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-139668628982352"]
Sep 08 11:49:36.170016 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:49:36.169992    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a5c4769_19ad_415f_b268_8f5a352ba651.slice/crio-9e44c566b844425025504b95662b537fa6b129fa2908da6d56e8a99d566330b5 WatchSource:0}: Error finding container 9e44c566b844425025504b95662b537fa6b129fa2908da6d56e8a99d566330b5: Status 404 returned error can't find the container with id 9e44c566b844425025504b95662b537fa6b129fa2908da6d56e8a99d566330b5
Sep 08 11:49:36.287173 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:36.287141    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:36.287173 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:36.287143    2561 scope.go:117] "RemoveContainer" containerID="8bdf9e1999f52a442177ef34edd2390b4d4256788fcddcd0a1a78a3b114b0f7b"
Sep 08 11:49:36.288582 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:36.288510    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-139668628982352" event={"ID":"4a5c4769-19ad-415f-b268-8f5a352ba651","Type":"ContainerStarted","Data":"9e44c566b844425025504b95662b537fa6b129fa2908da6d56e8a99d566330b5"}
Sep 08 11:49:37.296611 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:37.296577    2561 generic.go:356] "Generic (PLEG): container finished" podID="4a5c4769-19ad-415f-b268-8f5a352ba651" containerID="dbeff067bfafc743c1705ec7556df96b9c6f3d74859283e5ed89fb2d6f84d69c" exitCode=0
Sep 08 11:49:37.297010 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:37.296620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-139668628982352" event={"ID":"4a5c4769-19ad-415f-b268-8f5a352ba651","Type":"ContainerDied","Data":"dbeff067bfafc743c1705ec7556df96b9c6f3d74859283e5ed89fb2d6f84d69c"}
Sep 08 11:49:38.432307 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:38.432282    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:38.454929 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:38.454902    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7437-139668628982352_4a5c4769-19ad-415f-b268-8f5a352ba651/test-curl-7437-139668628982352/0.log"
Sep 08 11:49:38.455166 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:38.455149    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzcz5\" (UniqueName: \"kubernetes.io/projected/4a5c4769-19ad-415f-b268-8f5a352ba651-kube-api-access-mzcz5\") pod \"4a5c4769-19ad-415f-b268-8f5a352ba651\" (UID: \"4a5c4769-19ad-415f-b268-8f5a352ba651\") "
Sep 08 11:49:38.457103 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:38.457081    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5c4769-19ad-415f-b268-8f5a352ba651-kube-api-access-mzcz5" (OuterVolumeSpecName: "kube-api-access-mzcz5") pod "4a5c4769-19ad-415f-b268-8f5a352ba651" (UID: "4a5c4769-19ad-415f-b268-8f5a352ba651"). InnerVolumeSpecName "kube-api-access-mzcz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:49:38.476801 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:38.476776    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7437-139668628982352"]
Sep 08 11:49:38.481135 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:38.481105    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7437-139668628982352"]
Sep 08 11:49:38.555667 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:38.555613    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzcz5\" (UniqueName: \"kubernetes.io/projected/4a5c4769-19ad-415f-b268-8f5a352ba651-kube-api-access-mzcz5\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:49:39.086699 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:39.086656    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:39.308523 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:39.308496    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e44c566b844425025504b95662b537fa6b129fa2908da6d56e8a99d566330b5"
Sep 08 11:49:39.308523 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:39.308516    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:39.318405 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:49:39.318372    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7437-139668628982352\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="4a5c4769-19ad-415f-b268-8f5a352ba651" pod="opendatahub/test-curl-7437-139668628982352"
Sep 08 11:49:39.424335 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:39.424262    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:39.488962 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:39.488917    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:39.961250 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:39.961212    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a5c4769-19ad-415f-b268-8f5a352ba651" path="/var/lib/kubelet/pods/4a5c4769-19ad-415f-b268-8f5a352ba651/volumes"
Sep 08 11:49:43.117548 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:43.117428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:43.216491 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:43.216439    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:44.273454 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:44.273416    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:44.374776 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:44.374746    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:44.935320 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:44.935281    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"]
Sep 08 11:49:44.935724 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:44.935706    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a5c4769-19ad-415f-b268-8f5a352ba651" containerName="test-curl-7437-139668628982352"
Sep 08 11:49:44.935825 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:44.935725    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a5c4769-19ad-415f-b268-8f5a352ba651" containerName="test-curl-7437-139668628982352"
Sep 08 11:49:44.935825 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:44.935819    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a5c4769-19ad-415f-b268-8f5a352ba651" containerName="test-curl-7437-139668628982352"
Sep 08 11:49:44.939009 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:44.938988    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"
Sep 08 11:49:44.941457 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:44.941437    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-63d6ac92-dockercfg-gvdgp\""
Sep 08 11:49:44.941563 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:44.941500    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-63d6ac92\""
Sep 08 11:49:44.950704 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:44.950681    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"]
Sep 08 11:49:44.980289 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:44.980266    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"]
Sep 08 11:49:44.983422 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:44.983407    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"
Sep 08 11:49:44.994245 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:44.994228    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"]
Sep 08 11:49:45.011766 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.011744    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbvk8\" (UniqueName: \"kubernetes.io/projected/22500d68-d701-4187-b4c2-e81d10e5a579-kube-api-access-lbvk8\") pod \"payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69\" (UID: \"22500d68-d701-4187-b4c2-e81d10e5a579\") " pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"
Sep 08 11:49:45.011860 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.011780    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22500d68-d701-4187-b4c2-e81d10e5a579-plugins-config-volume\") pod \"payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69\" (UID: \"22500d68-d701-4187-b4c2-e81d10e5a579\") " pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"
Sep 08 11:49:45.011860 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.011807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vghj8\" (UniqueName: \"kubernetes.io/projected/9ddbd1b6-3751-4325-8b81-91d2acd4388f-kube-api-access-vghj8\") pod \"payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v\" (UID: \"9ddbd1b6-3751-4325-8b81-91d2acd4388f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"
Sep 08 11:49:45.011974 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.011885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ddbd1b6-3751-4325-8b81-91d2acd4388f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v\" (UID: \"9ddbd1b6-3751-4325-8b81-91d2acd4388f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"
Sep 08 11:49:45.112684 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.112650    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbvk8\" (UniqueName: \"kubernetes.io/projected/22500d68-d701-4187-b4c2-e81d10e5a579-kube-api-access-lbvk8\") pod \"payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69\" (UID: \"22500d68-d701-4187-b4c2-e81d10e5a579\") " pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"
Sep 08 11:49:45.112859 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.112693    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22500d68-d701-4187-b4c2-e81d10e5a579-plugins-config-volume\") pod \"payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69\" (UID: \"22500d68-d701-4187-b4c2-e81d10e5a579\") " pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"
Sep 08 11:49:45.112859 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.112736    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vghj8\" (UniqueName: \"kubernetes.io/projected/9ddbd1b6-3751-4325-8b81-91d2acd4388f-kube-api-access-vghj8\") pod \"payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v\" (UID: \"9ddbd1b6-3751-4325-8b81-91d2acd4388f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"
Sep 08 11:49:45.112859 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.112787    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ddbd1b6-3751-4325-8b81-91d2acd4388f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v\" (UID: \"9ddbd1b6-3751-4325-8b81-91d2acd4388f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"
Sep 08 11:49:45.113555 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.113521    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ddbd1b6-3751-4325-8b81-91d2acd4388f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v\" (UID: \"9ddbd1b6-3751-4325-8b81-91d2acd4388f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"
Sep 08 11:49:45.113673 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.113567    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22500d68-d701-4187-b4c2-e81d10e5a579-plugins-config-volume\") pod \"payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69\" (UID: \"22500d68-d701-4187-b4c2-e81d10e5a579\") " pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"
Sep 08 11:49:45.122358 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.122329    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbvk8\" (UniqueName: \"kubernetes.io/projected/22500d68-d701-4187-b4c2-e81d10e5a579-kube-api-access-lbvk8\") pod \"payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69\" (UID: \"22500d68-d701-4187-b4c2-e81d10e5a579\") " pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"
Sep 08 11:49:45.123237 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.123216    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vghj8\" (UniqueName: \"kubernetes.io/projected/9ddbd1b6-3751-4325-8b81-91d2acd4388f-kube-api-access-vghj8\") pod \"payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v\" (UID: \"9ddbd1b6-3751-4325-8b81-91d2acd4388f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"
Sep 08 11:49:45.249814 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.249783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"
Sep 08 11:49:45.292175 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.292146    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"
Sep 08 11:49:45.526588 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.526562    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"]
Sep 08 11:49:45.527833 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:49:45.527807    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22500d68_d701_4187_b4c2_e81d10e5a579.slice/crio-e2ce9f218e1f928790da5fa91ed8f46a97d180c5df642759483af6bd84ce05f1 WatchSource:0}: Error finding container e2ce9f218e1f928790da5fa91ed8f46a97d180c5df642759483af6bd84ce05f1: Status 404 returned error can't find the container with id e2ce9f218e1f928790da5fa91ed8f46a97d180c5df642759483af6bd84ce05f1
Sep 08 11:49:45.573822 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:45.573794    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"]
Sep 08 11:49:45.576352 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:49:45.576327    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ddbd1b6_3751_4325_8b81_91d2acd4388f.slice/crio-850ae1de6224e163751b35bd178625ae21aef94fc00ae9b86dfc321e146522c7 WatchSource:0}: Error finding container 850ae1de6224e163751b35bd178625ae21aef94fc00ae9b86dfc321e146522c7: Status 404 returned error can't find the container with id 850ae1de6224e163751b35bd178625ae21aef94fc00ae9b86dfc321e146522c7
Sep 08 11:49:46.350829 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:46.350795    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69" event={"ID":"22500d68-d701-4187-b4c2-e81d10e5a579","Type":"ContainerStarted","Data":"5491e54ab902e1c2f8b5137e4cd990c01e793d25b88abb03ca08f3e60a6d5106"}
Sep 08 11:49:46.351295 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:46.350850    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69" event={"ID":"22500d68-d701-4187-b4c2-e81d10e5a579","Type":"ContainerStarted","Data":"e2ce9f218e1f928790da5fa91ed8f46a97d180c5df642759483af6bd84ce05f1"}
Sep 08 11:49:46.351295 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:46.350904    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"
Sep 08 11:49:46.352506 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:46.352485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v" event={"ID":"9ddbd1b6-3751-4325-8b81-91d2acd4388f","Type":"ContainerStarted","Data":"5ea6b60e0f91bc7e0d3bb60fab6546a80d20714002221c4be8d22b488953126a"}
Sep 08 11:49:46.352570 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:46.352521    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v" event={"ID":"9ddbd1b6-3751-4325-8b81-91d2acd4388f","Type":"ContainerStarted","Data":"850ae1de6224e163751b35bd178625ae21aef94fc00ae9b86dfc321e146522c7"}
Sep 08 11:49:46.352570 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:46.352549    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"
Sep 08 11:49:46.370760 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:46.370710    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69" podStartSLOduration=2.370693504 podStartE2EDuration="2.370693504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:49:46.368092866 +0000 UTC m=+1192.934853407" watchObservedRunningTime="2026-09-08 11:49:46.370693504 +0000 UTC m=+1192.937454045"
Sep 08 11:49:46.376932 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:46.376905    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:46.387083 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:46.387044    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v" podStartSLOduration=2.387030881 podStartE2EDuration="2.387030881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:49:46.385417808 +0000 UTC m=+1192.952178350" watchObservedRunningTime="2026-09-08 11:49:46.387030881 +0000 UTC m=+1192.953791424"
Sep 08 11:49:46.511186 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:46.511138    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:49:54.045303 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:54.045267    2561 scope.go:117] "RemoveContainer" containerID="41f2fb64a54a49b74f624988a5aad2192f592f7119feb3bab44a4ef04eceb051"
Sep 08 11:49:54.053521 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:54.053499    2561 scope.go:117] "RemoveContainer" containerID="601c688afc08091b786540cbe8566a3d8d09f409f1d414c271bd673f907493eb"
Sep 08 11:49:54.061057 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:54.061039    2561 scope.go:117] "RemoveContainer" containerID="5acbdb09b425558986af66ae33706e0b3fa2196e02c2b6a1ca20f44701579a6a"
Sep 08 11:49:54.068141 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:54.068121    2561 scope.go:117] "RemoveContainer" containerID="40d41afc2cafe286b0c1b4beeaf35e75b19b339ee4b35f596a169ea6d9d6eb72"
Sep 08 11:49:57.359872 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:57.359841    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"
Sep 08 11:49:57.360318 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:49:57.359910    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"
Sep 08 11:50:07.992462 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:07.992428    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:50:28.079716 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:28.079669    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:50:31.050892 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:31.050849    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:50:35.104696 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:35.104661    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:50:35.171590 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:35.171559    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:50:39.916386 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:39.916351    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7440-e96120"]
Sep 08 11:50:39.919721 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:39.919700    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7440-e96120"
Sep 08 11:50:39.922731 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:39.922711    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zbjsb\""
Sep 08 11:50:39.928789 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:39.928767    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7440-e96120"]
Sep 08 11:50:39.973321 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:39.973295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8sqw\" (UniqueName: \"kubernetes.io/projected/aac2702f-c60c-4adc-b2b7-f0d4d3d01953-kube-api-access-l8sqw\") pod \"mt-curl-7440-e96120\" (UID: \"aac2702f-c60c-4adc-b2b7-f0d4d3d01953\") " pod="opendatahub/mt-curl-7440-e96120"
Sep 08 11:50:40.074273 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:40.074244    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8sqw\" (UniqueName: \"kubernetes.io/projected/aac2702f-c60c-4adc-b2b7-f0d4d3d01953-kube-api-access-l8sqw\") pod \"mt-curl-7440-e96120\" (UID: \"aac2702f-c60c-4adc-b2b7-f0d4d3d01953\") " pod="opendatahub/mt-curl-7440-e96120"
Sep 08 11:50:40.082887 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:40.082862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8sqw\" (UniqueName: \"kubernetes.io/projected/aac2702f-c60c-4adc-b2b7-f0d4d3d01953-kube-api-access-l8sqw\") pod \"mt-curl-7440-e96120\" (UID: \"aac2702f-c60c-4adc-b2b7-f0d4d3d01953\") " pod="opendatahub/mt-curl-7440-e96120"
Sep 08 11:50:40.229547 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:40.229517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7440-e96120"
Sep 08 11:50:40.499177 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:40.499146    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7440-e96120"]
Sep 08 11:50:40.501221 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:50:40.501193    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaac2702f_c60c_4adc_b2b7_f0d4d3d01953.slice/crio-7ad95ce16db6874b107e2e192f110e1704c66a5315eb0d405f74e971a61bee1b WatchSource:0}: Error finding container 7ad95ce16db6874b107e2e192f110e1704c66a5315eb0d405f74e971a61bee1b: Status 404 returned error can't find the container with id 7ad95ce16db6874b107e2e192f110e1704c66a5315eb0d405f74e971a61bee1b
Sep 08 11:50:40.639149 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:40.639115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7440-e96120" event={"ID":"aac2702f-c60c-4adc-b2b7-f0d4d3d01953","Type":"ContainerStarted","Data":"7ad95ce16db6874b107e2e192f110e1704c66a5315eb0d405f74e971a61bee1b"}
Sep 08 11:50:41.645289 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:41.645253    2561 generic.go:356] "Generic (PLEG): container finished" podID="aac2702f-c60c-4adc-b2b7-f0d4d3d01953" containerID="e714dc8a6dc25377e0b74c37414af92191afef1501415b7720f599cb68064708" exitCode=0
Sep 08 11:50:41.645690 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:41.645344    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7440-e96120" event={"ID":"aac2702f-c60c-4adc-b2b7-f0d4d3d01953","Type":"ContainerDied","Data":"e714dc8a6dc25377e0b74c37414af92191afef1501415b7720f599cb68064708"}
Sep 08 11:50:42.781507 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:42.781484    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7440-e96120"
Sep 08 11:50:42.806515 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:42.806481    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7440-e96120_aac2702f-c60c-4adc-b2b7-f0d4d3d01953/mt-curl-7440-e96120/0.log"
Sep 08 11:50:42.829928 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:42.829904    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7440-e96120"]
Sep 08 11:50:42.834376 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:42.834353    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7440-e96120"]
Sep 08 11:50:42.897041 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:42.897012    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8sqw\" (UniqueName: \"kubernetes.io/projected/aac2702f-c60c-4adc-b2b7-f0d4d3d01953-kube-api-access-l8sqw\") pod \"aac2702f-c60c-4adc-b2b7-f0d4d3d01953\" (UID: \"aac2702f-c60c-4adc-b2b7-f0d4d3d01953\") "
Sep 08 11:50:42.899068 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:42.899042    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac2702f-c60c-4adc-b2b7-f0d4d3d01953-kube-api-access-l8sqw" (OuterVolumeSpecName: "kube-api-access-l8sqw") pod "aac2702f-c60c-4adc-b2b7-f0d4d3d01953" (UID: "aac2702f-c60c-4adc-b2b7-f0d4d3d01953"). InnerVolumeSpecName "kube-api-access-l8sqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:50:42.998891 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:42.998860    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8sqw\" (UniqueName: \"kubernetes.io/projected/aac2702f-c60c-4adc-b2b7-f0d4d3d01953-kube-api-access-l8sqw\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:50:43.027743 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:43.027708    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:50:43.657091 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:43.657065    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ad95ce16db6874b107e2e192f110e1704c66a5315eb0d405f74e971a61bee1b"
Sep 08 11:50:43.657091 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:43.657074    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7440-e96120"
Sep 08 11:50:43.667551 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:50:43.667515    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7440-e96120\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="aac2702f-c60c-4adc-b2b7-f0d4d3d01953" pod="opendatahub/mt-curl-7440-e96120"
Sep 08 11:50:43.961155 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:50:43.961111    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7440-e96120\" is forbidden: User \"system:node:ip-10-0-139-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-41.ec2.internal' and this object" podUID="aac2702f-c60c-4adc-b2b7-f0d4d3d01953" pod="opendatahub/mt-curl-7440-e96120"
Sep 08 11:50:43.962387 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:43.962365    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac2702f-c60c-4adc-b2b7-f0d4d3d01953" path="/var/lib/kubelet/pods/aac2702f-c60c-4adc-b2b7-f0d4d3d01953/volumes"
Sep 08 11:50:53.103131 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:53.103050    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"]
Sep 08 11:50:53.146161 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:53.146096    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:50:54.107125 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:54.107092    2561 scope.go:117] "RemoveContainer" containerID="042bda6d2466c80a1958bebee18bdc89d8d5e4edcfd61c6f1c59841dfcd2e6dd"
Sep 08 11:50:54.115408 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:54.115388    2561 scope.go:117] "RemoveContainer" containerID="8b446ad5f83786582758f384f073cc6fcba4bfa381ba17135cef189f3045f86e"
Sep 08 11:50:54.123074 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:54.123051    2561 scope.go:117] "RemoveContainer" containerID="0dda68215d53bcb36933c808f16aba9094dc48c43ae5d1142fe3c920a4ea5404"
Sep 08 11:50:55.254596 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:50:55.254554    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:51:05.022640 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.022601    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"]
Sep 08 11:51:05.023198 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.023105    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac2702f-c60c-4adc-b2b7-f0d4d3d01953" containerName="mt-curl-7440-e96120"
Sep 08 11:51:05.023198 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.023124    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac2702f-c60c-4adc-b2b7-f0d4d3d01953" containerName="mt-curl-7440-e96120"
Sep 08 11:51:05.023323 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.023233    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac2702f-c60c-4adc-b2b7-f0d4d3d01953" containerName="mt-curl-7440-e96120"
Sep 08 11:51:05.027575 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.027555    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.030228 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.030210    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-838a68\"/\"sub-test-model-838a68-kserve-self-signed-certs\""
Sep 08 11:51:05.036145 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.036122    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"]
Sep 08 11:51:05.065170 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.065141    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:51:05.086274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.086242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-model-cache\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.086399 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.086278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czgmv\" (UniqueName: \"kubernetes.io/projected/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-kube-api-access-czgmv\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.086399 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.086303    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-tls-certs\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.086399 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.086320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-home\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.086399 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.086397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-dshm\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.086585 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.086422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-tmp-dir\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.186861 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.186829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-dshm\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.187056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.186888    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-tmp-dir\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.187056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.186923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-model-cache\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.187056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.186955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czgmv\" (UniqueName: \"kubernetes.io/projected/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-kube-api-access-czgmv\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.187056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.186993    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-tls-certs\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.187289 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.187110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-home\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.187367 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.187338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-model-cache\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.187367 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.187353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-tmp-dir\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.187575 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.187555    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-home\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.189116 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.189094    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-dshm\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.189513 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.189492    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-tls-certs\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.196048 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.196026    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czgmv\" (UniqueName: \"kubernetes.io/projected/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-kube-api-access-czgmv\") pod \"sub-test-model-838a68-kserve-66bbf854ff-nh75w\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") " pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.337584 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.337491    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:05.608898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.608871    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"]
Sep 08 11:51:05.610480 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:51:05.610442    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb03f8b9c_30ac_4e56_b530_2dcac564ed3b.slice/crio-6b61518f028aa5f68eacb305c9f03e3b5082e2ca5ea4e57826b63cfa929d441a WatchSource:0}: Error finding container 6b61518f028aa5f68eacb305c9f03e3b5082e2ca5ea4e57826b63cfa929d441a: Status 404 returned error can't find the container with id 6b61518f028aa5f68eacb305c9f03e3b5082e2ca5ea4e57826b63cfa929d441a
Sep 08 11:51:05.772976 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.772941    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w" event={"ID":"b03f8b9c-30ac-4e56-b530-2dcac564ed3b","Type":"ContainerStarted","Data":"3e51d0fd6c377e189d306958d68bb62c7412f1ac10db12eac515b74ff1713165"}
Sep 08 11:51:05.772976 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.772981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w" event={"ID":"b03f8b9c-30ac-4e56-b530-2dcac564ed3b","Type":"ContainerStarted","Data":"6b61518f028aa5f68eacb305c9f03e3b5082e2ca5ea4e57826b63cfa929d441a"}
Sep 08 11:51:05.792355 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:05.792310    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w" podStartSLOduration=0.792296986 podStartE2EDuration="792.296986ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:51:05.790157847 +0000 UTC m=+1272.356918392" watchObservedRunningTime="2026-09-08 11:51:05.792296986 +0000 UTC m=+1272.359057556"
Sep 08 11:51:07.832924 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:07.832892    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz"]
Sep 08 11:51:07.836257 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:07.836241    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz"
Sep 08 11:51:07.839082 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:07.839060    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:51:07.839238 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:07.839147    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:51:07.846567 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:07.846537    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz"]
Sep 08 11:51:07.908914 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:07.908880    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6bd5b2-4590-4303-909d-fef1336cc3ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz\" (UID: \"dd6bd5b2-4590-4303-909d-fef1336cc3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz"
Sep 08 11:51:07.960166 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:07.960132    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:51:08.009801 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.009768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6bd5b2-4590-4303-909d-fef1336cc3ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz\" (UID: \"dd6bd5b2-4590-4303-909d-fef1336cc3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz"
Sep 08 11:51:08.010357 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.010339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6bd5b2-4590-4303-909d-fef1336cc3ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz\" (UID: \"dd6bd5b2-4590-4303-909d-fef1336cc3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz"
Sep 08 11:51:08.122660 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.122557    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q" podUID="b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37" containerName="main" containerID="cri-o://cecfa448fce639b532be8342d09f995552f4a833ed76bb862ae802909a6f4c90" gracePeriod=45
Sep 08 11:51:08.148054 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.148026    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz"
Sep 08 11:51:08.368453 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.368425    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:51:08.421396 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.421312    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz"]
Sep 08 11:51:08.424452 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:51:08.424423    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6bd5b2_4590_4303_909d_fef1336cc3ef.slice/crio-acb4a5a98b3be9d2505d2de2acb782bcb23e32356a2c928d9f371da950a6f942 WatchSource:0}: Error finding container acb4a5a98b3be9d2505d2de2acb782bcb23e32356a2c928d9f371da950a6f942: Status 404 returned error can't find the container with id acb4a5a98b3be9d2505d2de2acb782bcb23e32356a2c928d9f371da950a6f942
Sep 08 11:51:08.513432 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.513408    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pt8f\" (UniqueName: \"kubernetes.io/projected/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-kube-api-access-9pt8f\") pod \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") "
Sep 08 11:51:08.513555 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.513536    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-tmp-dir\") pod \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") "
Sep 08 11:51:08.513615 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.513570    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-model-cache\") pod \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") "
Sep 08 11:51:08.513615 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.513607    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-home\") pod \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") "
Sep 08 11:51:08.513727 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.513644    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-dshm\") pod \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") "
Sep 08 11:51:08.513727 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.513683    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-tls-certs\") pod \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\" (UID: \"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37\") "
Sep 08 11:51:08.513844 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.513816    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37" (UID: "b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:51:08.514028 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.513988    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-tmp-dir\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:51:08.514028 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.513986    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-home" (OuterVolumeSpecName: "home") pod "b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37" (UID: "b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:51:08.514152 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.514021    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37" (UID: "b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:51:08.515680 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.515658    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-kube-api-access-9pt8f" (OuterVolumeSpecName: "kube-api-access-9pt8f") pod "b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37" (UID: "b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37"). InnerVolumeSpecName "kube-api-access-9pt8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:51:08.515768 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.515728    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-dshm" (OuterVolumeSpecName: "dshm") pod "b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37" (UID: "b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:51:08.515768 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.515736    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37" (UID: "b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:51:08.614555 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.614529    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-model-cache\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:51:08.614555 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.614550    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-home\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:51:08.614555 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.614559    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-dshm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:51:08.614735 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.614566    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-tls-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:51:08.614735 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.614574    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pt8f\" (UniqueName: \"kubernetes.io/projected/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37-kube-api-access-9pt8f\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:51:08.789281 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.789232    2561 generic.go:356] "Generic (PLEG): container finished" podID="b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37" containerID="cecfa448fce639b532be8342d09f995552f4a833ed76bb862ae802909a6f4c90" exitCode=0
Sep 08 11:51:08.789281 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.789279    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"
Sep 08 11:51:08.789743 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.789313    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q" event={"ID":"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37","Type":"ContainerDied","Data":"cecfa448fce639b532be8342d09f995552f4a833ed76bb862ae802909a6f4c90"}
Sep 08 11:51:08.789743 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.789354    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q" event={"ID":"b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37","Type":"ContainerDied","Data":"1f2bd6d17d260a58381b5aa220661c6c7bd886c21852c977cc433877476639ab"}
Sep 08 11:51:08.789743 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.789372    2561 scope.go:117] "RemoveContainer" containerID="cecfa448fce639b532be8342d09f995552f4a833ed76bb862ae802909a6f4c90"
Sep 08 11:51:08.791119 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.791092    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz" event={"ID":"dd6bd5b2-4590-4303-909d-fef1336cc3ef","Type":"ContainerStarted","Data":"62a7e9620dfa87130f0f2a24cbac0f6656837057571142cd15dacd95ca2d6e69"}
Sep 08 11:51:08.791247 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.791150    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz" event={"ID":"dd6bd5b2-4590-4303-909d-fef1336cc3ef","Type":"ContainerStarted","Data":"acb4a5a98b3be9d2505d2de2acb782bcb23e32356a2c928d9f371da950a6f942"}
Sep 08 11:51:08.798979 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.798960    2561 scope.go:117] "RemoveContainer" containerID="cecfa448fce639b532be8342d09f995552f4a833ed76bb862ae802909a6f4c90"
Sep 08 11:51:08.799258 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:51:08.799237    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cecfa448fce639b532be8342d09f995552f4a833ed76bb862ae802909a6f4c90\": container with ID starting with cecfa448fce639b532be8342d09f995552f4a833ed76bb862ae802909a6f4c90 not found: ID does not exist" containerID="cecfa448fce639b532be8342d09f995552f4a833ed76bb862ae802909a6f4c90"
Sep 08 11:51:08.799319 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.799269    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecfa448fce639b532be8342d09f995552f4a833ed76bb862ae802909a6f4c90"} err="failed to get container status \"cecfa448fce639b532be8342d09f995552f4a833ed76bb862ae802909a6f4c90\": rpc error: code = NotFound desc = could not find container \"cecfa448fce639b532be8342d09f995552f4a833ed76bb862ae802909a6f4c90\": container with ID starting with cecfa448fce639b532be8342d09f995552f4a833ed76bb862ae802909a6f4c90 not found: ID does not exist"
Sep 08 11:51:08.813418 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.813380    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz" podStartSLOduration=1.813366246 podStartE2EDuration="1.813366246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:51:08.811360465 +0000 UTC m=+1275.378121011" watchObservedRunningTime="2026-09-08 11:51:08.813366246 +0000 UTC m=+1275.380126787"
Sep 08 11:51:08.826481 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.826444    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"]
Sep 08 11:51:08.830892 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:08.830872    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/auth-test-model-838a68-kserve-58bdcdf475-vsx8q"]
Sep 08 11:51:09.798806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:09.798774    2561 generic.go:356] "Generic (PLEG): container finished" podID="dd6bd5b2-4590-4303-909d-fef1336cc3ef" containerID="62a7e9620dfa87130f0f2a24cbac0f6656837057571142cd15dacd95ca2d6e69" exitCode=0
Sep 08 11:51:09.799169 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:09.798838    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz" event={"ID":"dd6bd5b2-4590-4303-909d-fef1336cc3ef","Type":"ContainerDied","Data":"62a7e9620dfa87130f0f2a24cbac0f6656837057571142cd15dacd95ca2d6e69"}
Sep 08 11:51:09.961835 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:09.961792    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37" path="/var/lib/kubelet/pods/b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37/volumes"
Sep 08 11:51:10.931448 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:10.931423    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz"
Sep 08 11:51:11.036905 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:11.036877    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6bd5b2-4590-4303-909d-fef1336cc3ef-maas-api-service-ca\") pod \"dd6bd5b2-4590-4303-909d-fef1336cc3ef\" (UID: \"dd6bd5b2-4590-4303-909d-fef1336cc3ef\") "
Sep 08 11:51:11.037203 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:11.037182    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6bd5b2-4590-4303-909d-fef1336cc3ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd6bd5b2-4590-4303-909d-fef1336cc3ef" (UID: "dd6bd5b2-4590-4303-909d-fef1336cc3ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:51:11.137924 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:11.137851    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6bd5b2-4590-4303-909d-fef1336cc3ef-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:51:11.809952 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:11.809924    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz"
Sep 08 11:51:11.809952 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:11.809945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz" event={"ID":"dd6bd5b2-4590-4303-909d-fef1336cc3ef","Type":"ContainerDied","Data":"acb4a5a98b3be9d2505d2de2acb782bcb23e32356a2c928d9f371da950a6f942"}
Sep 08 11:51:11.810151 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:11.809983    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acb4a5a98b3be9d2505d2de2acb782bcb23e32356a2c928d9f371da950a6f942"
Sep 08 11:51:14.851219 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:14.851185    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:51:14.923305 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:14.923271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:51:15.338584 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:15.338552    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:15.338747 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:15.338604    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:15.351155 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:15.351131    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:15.842402 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:15.842374    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:51:23.488800 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.488738    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:51:23.506845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.506802    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"]
Sep 08 11:51:23.507056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.507023    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69" podUID="22500d68-d701-4187-b4c2-e81d10e5a579" containerName="payload-processing" containerID="cri-o://5491e54ab902e1c2f8b5137e4cd990c01e793d25b88abb03ca08f3e60a6d5106" gracePeriod=30
Sep 08 11:51:23.508968 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.508939    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"]
Sep 08 11:51:23.509148 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.509121    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v" podUID="9ddbd1b6-3751-4325-8b81-91d2acd4388f" containerName="payload-pre-processing" containerID="cri-o://5ea6b60e0f91bc7e0d3bb60fab6546a80d20714002221c4be8d22b488953126a" gracePeriod=30
Sep 08 11:51:23.591128 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.591084    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:51:23.689752 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.689725    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:51:23.696750 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.696725    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz"]
Sep 08 11:51:23.699909 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.699885    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-63d6ac92-k4qrz"]
Sep 08 11:51:23.782002 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.781983    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"
Sep 08 11:51:23.784747 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.784729    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"
Sep 08 11:51:23.791457 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.791436    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:51:23.856297 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.856267    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22500d68-d701-4187-b4c2-e81d10e5a579-plugins-config-volume\") pod \"22500d68-d701-4187-b4c2-e81d10e5a579\" (UID: \"22500d68-d701-4187-b4c2-e81d10e5a579\") "
Sep 08 11:51:23.856436 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.856302    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ddbd1b6-3751-4325-8b81-91d2acd4388f-plugins-config-volume\") pod \"9ddbd1b6-3751-4325-8b81-91d2acd4388f\" (UID: \"9ddbd1b6-3751-4325-8b81-91d2acd4388f\") "
Sep 08 11:51:23.856436 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.856337    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbvk8\" (UniqueName: \"kubernetes.io/projected/22500d68-d701-4187-b4c2-e81d10e5a579-kube-api-access-lbvk8\") pod \"22500d68-d701-4187-b4c2-e81d10e5a579\" (UID: \"22500d68-d701-4187-b4c2-e81d10e5a579\") "
Sep 08 11:51:23.856436 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.856368    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vghj8\" (UniqueName: \"kubernetes.io/projected/9ddbd1b6-3751-4325-8b81-91d2acd4388f-kube-api-access-vghj8\") pod \"9ddbd1b6-3751-4325-8b81-91d2acd4388f\" (UID: \"9ddbd1b6-3751-4325-8b81-91d2acd4388f\") "
Sep 08 11:51:23.856638 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.856614    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22500d68-d701-4187-b4c2-e81d10e5a579-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22500d68-d701-4187-b4c2-e81d10e5a579" (UID: "22500d68-d701-4187-b4c2-e81d10e5a579"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:51:23.856697 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.856667    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ddbd1b6-3751-4325-8b81-91d2acd4388f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ddbd1b6-3751-4325-8b81-91d2acd4388f" (UID: "9ddbd1b6-3751-4325-8b81-91d2acd4388f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:51:23.858427 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.858399    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22500d68-d701-4187-b4c2-e81d10e5a579-kube-api-access-lbvk8" (OuterVolumeSpecName: "kube-api-access-lbvk8") pod "22500d68-d701-4187-b4c2-e81d10e5a579" (UID: "22500d68-d701-4187-b4c2-e81d10e5a579"). InnerVolumeSpecName "kube-api-access-lbvk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:51:23.858527 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.858422    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ddbd1b6-3751-4325-8b81-91d2acd4388f-kube-api-access-vghj8" (OuterVolumeSpecName: "kube-api-access-vghj8") pod "9ddbd1b6-3751-4325-8b81-91d2acd4388f" (UID: "9ddbd1b6-3751-4325-8b81-91d2acd4388f"). InnerVolumeSpecName "kube-api-access-vghj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:51:23.877217 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.877192    2561 generic.go:356] "Generic (PLEG): container finished" podID="22500d68-d701-4187-b4c2-e81d10e5a579" containerID="5491e54ab902e1c2f8b5137e4cd990c01e793d25b88abb03ca08f3e60a6d5106" exitCode=0
Sep 08 11:51:23.877289 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.877236    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"
Sep 08 11:51:23.877289 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.877264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69" event={"ID":"22500d68-d701-4187-b4c2-e81d10e5a579","Type":"ContainerDied","Data":"5491e54ab902e1c2f8b5137e4cd990c01e793d25b88abb03ca08f3e60a6d5106"}
Sep 08 11:51:23.877365 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.877312    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69" event={"ID":"22500d68-d701-4187-b4c2-e81d10e5a579","Type":"ContainerDied","Data":"e2ce9f218e1f928790da5fa91ed8f46a97d180c5df642759483af6bd84ce05f1"}
Sep 08 11:51:23.877365 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.877335    2561 scope.go:117] "RemoveContainer" containerID="5491e54ab902e1c2f8b5137e4cd990c01e793d25b88abb03ca08f3e60a6d5106"
Sep 08 11:51:23.878712 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.878692    2561 generic.go:356] "Generic (PLEG): container finished" podID="9ddbd1b6-3751-4325-8b81-91d2acd4388f" containerID="5ea6b60e0f91bc7e0d3bb60fab6546a80d20714002221c4be8d22b488953126a" exitCode=0
Sep 08 11:51:23.878804 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.878723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v" event={"ID":"9ddbd1b6-3751-4325-8b81-91d2acd4388f","Type":"ContainerDied","Data":"5ea6b60e0f91bc7e0d3bb60fab6546a80d20714002221c4be8d22b488953126a"}
Sep 08 11:51:23.878804 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.878743    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"
Sep 08 11:51:23.878804 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.878759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v" event={"ID":"9ddbd1b6-3751-4325-8b81-91d2acd4388f","Type":"ContainerDied","Data":"850ae1de6224e163751b35bd178625ae21aef94fc00ae9b86dfc321e146522c7"}
Sep 08 11:51:23.889039 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.889016    2561 scope.go:117] "RemoveContainer" containerID="5491e54ab902e1c2f8b5137e4cd990c01e793d25b88abb03ca08f3e60a6d5106"
Sep 08 11:51:23.889425 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:51:23.889393    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5491e54ab902e1c2f8b5137e4cd990c01e793d25b88abb03ca08f3e60a6d5106\": container with ID starting with 5491e54ab902e1c2f8b5137e4cd990c01e793d25b88abb03ca08f3e60a6d5106 not found: ID does not exist" containerID="5491e54ab902e1c2f8b5137e4cd990c01e793d25b88abb03ca08f3e60a6d5106"
Sep 08 11:51:23.889561 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.889441    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5491e54ab902e1c2f8b5137e4cd990c01e793d25b88abb03ca08f3e60a6d5106"} err="failed to get container status \"5491e54ab902e1c2f8b5137e4cd990c01e793d25b88abb03ca08f3e60a6d5106\": rpc error: code = NotFound desc = could not find container \"5491e54ab902e1c2f8b5137e4cd990c01e793d25b88abb03ca08f3e60a6d5106\": container with ID starting with 5491e54ab902e1c2f8b5137e4cd990c01e793d25b88abb03ca08f3e60a6d5106 not found: ID does not exist"
Sep 08 11:51:23.889561 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.889482    2561 scope.go:117] "RemoveContainer" containerID="5ea6b60e0f91bc7e0d3bb60fab6546a80d20714002221c4be8d22b488953126a"
Sep 08 11:51:23.898614 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.898592    2561 scope.go:117] "RemoveContainer" containerID="5ea6b60e0f91bc7e0d3bb60fab6546a80d20714002221c4be8d22b488953126a"
Sep 08 11:51:23.898875 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:51:23.898855    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea6b60e0f91bc7e0d3bb60fab6546a80d20714002221c4be8d22b488953126a\": container with ID starting with 5ea6b60e0f91bc7e0d3bb60fab6546a80d20714002221c4be8d22b488953126a not found: ID does not exist" containerID="5ea6b60e0f91bc7e0d3bb60fab6546a80d20714002221c4be8d22b488953126a"
Sep 08 11:51:23.898915 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.898882    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea6b60e0f91bc7e0d3bb60fab6546a80d20714002221c4be8d22b488953126a"} err="failed to get container status \"5ea6b60e0f91bc7e0d3bb60fab6546a80d20714002221c4be8d22b488953126a\": rpc error: code = NotFound desc = could not find container \"5ea6b60e0f91bc7e0d3bb60fab6546a80d20714002221c4be8d22b488953126a\": container with ID starting with 5ea6b60e0f91bc7e0d3bb60fab6546a80d20714002221c4be8d22b488953126a not found: ID does not exist"
Sep 08 11:51:23.904565 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.904546    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"]
Sep 08 11:51:23.909103 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.909083    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-63d6ac92-5f9969f8cc-bxd69"]
Sep 08 11:51:23.921683 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.921664    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"]
Sep 08 11:51:23.926172 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.926154    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-63d6ac92-f977c9f9d-kgm7v"]
Sep 08 11:51:23.957251 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.957228    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22500d68-d701-4187-b4c2-e81d10e5a579-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:51:23.957251 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.957253    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ddbd1b6-3751-4325-8b81-91d2acd4388f-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:51:23.957388 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.957263    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbvk8\" (UniqueName: \"kubernetes.io/projected/22500d68-d701-4187-b4c2-e81d10e5a579-kube-api-access-lbvk8\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:51:23.957388 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.957271    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vghj8\" (UniqueName: \"kubernetes.io/projected/9ddbd1b6-3751-4325-8b81-91d2acd4388f-kube-api-access-vghj8\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:51:23.961354 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.961332    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22500d68-d701-4187-b4c2-e81d10e5a579" path="/var/lib/kubelet/pods/22500d68-d701-4187-b4c2-e81d10e5a579/volumes"
Sep 08 11:51:23.961754 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.961740    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ddbd1b6-3751-4325-8b81-91d2acd4388f" path="/var/lib/kubelet/pods/9ddbd1b6-3751-4325-8b81-91d2acd4388f/volumes"
Sep 08 11:51:23.962064 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:23.962052    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd6bd5b2-4590-4303-909d-fef1336cc3ef" path="/var/lib/kubelet/pods/dd6bd5b2-4590-4303-909d-fef1336cc3ef/volumes"
Sep 08 11:51:26.628189 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.628156    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"]
Sep 08 11:51:26.628559 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.628525    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37" containerName="main"
Sep 08 11:51:26.628559 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.628537    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37" containerName="main"
Sep 08 11:51:26.628559 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.628548    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd6bd5b2-4590-4303-909d-fef1336cc3ef" containerName="revoke-keys"
Sep 08 11:51:26.628559 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.628553    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd6bd5b2-4590-4303-909d-fef1336cc3ef" containerName="revoke-keys"
Sep 08 11:51:26.628698 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.628562    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22500d68-d701-4187-b4c2-e81d10e5a579" containerName="payload-processing"
Sep 08 11:51:26.628698 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.628567    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="22500d68-d701-4187-b4c2-e81d10e5a579" containerName="payload-processing"
Sep 08 11:51:26.628698 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.628574    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ddbd1b6-3751-4325-8b81-91d2acd4388f" containerName="payload-pre-processing"
Sep 08 11:51:26.628698 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.628580    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ddbd1b6-3751-4325-8b81-91d2acd4388f" containerName="payload-pre-processing"
Sep 08 11:51:26.628698 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.628641    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7f2eefa-77ff-4e77-9d7c-5ff716bc6e37" containerName="main"
Sep 08 11:51:26.628698 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.628648    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ddbd1b6-3751-4325-8b81-91d2acd4388f" containerName="payload-pre-processing"
Sep 08 11:51:26.628698 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.628655    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="22500d68-d701-4187-b4c2-e81d10e5a579" containerName="payload-processing"
Sep 08 11:51:26.628698 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.628661    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd6bd5b2-4590-4303-909d-fef1336cc3ef" containerName="revoke-keys"
Sep 08 11:51:26.632849 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.632832    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.636743 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.636721    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bb9b64\"/\"kube-root-ca.crt\""
Sep 08 11:51:26.636968 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.636944    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bb9b64\"/\"openshift-service-ca.crt\""
Sep 08 11:51:26.637072 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.636984    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bb9b64\"/\"default-dockercfg-bbswc\""
Sep 08 11:51:26.637072 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.636984    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bb9b64\"/\"sub-test-model-bb9b64-kserve-self-signed-certs\""
Sep 08 11:51:26.642963 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.642942    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"]
Sep 08 11:51:26.647835 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.647815    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:51:26.676685 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.676661    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-dshm\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.676777 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.676694    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-tmp-dir\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.676777 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.676717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-home\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.676777 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.676734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-model-cache\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.676880 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.676820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-tls-certs\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.676880 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.676861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfmwx\" (UniqueName: \"kubernetes.io/projected/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-kube-api-access-jfmwx\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.777608 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.777579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-home\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.777608 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.777609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-model-cache\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.777800 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.777642    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-tls-certs\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.777800 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.777665    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfmwx\" (UniqueName: \"kubernetes.io/projected/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-kube-api-access-jfmwx\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.777800 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.777712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-dshm\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.777800 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.777737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-tmp-dir\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.778057 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.778033    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-model-cache\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.778116 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.778041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-home\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.778179 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.778133    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-tmp-dir\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.779853 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.779829    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-dshm\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.780067 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.780047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-tls-certs\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.786579 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.786557    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfmwx\" (UniqueName: \"kubernetes.io/projected/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-kube-api-access-jfmwx\") pod \"sub-test-model-bb9b64-kserve-6c86789b85-gqxgk\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:26.901462 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.901382    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:51:26.943367 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:26.943342    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:27.219455 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:27.219431    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"]
Sep 08 11:51:27.221646 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:51:27.221621    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cfa9f4f_ee2c_461c_9aa8_e0f3dc65b1ac.slice/crio-dfa29b9759cdc20c876e0dd1c5440c33b674aeb2a327867b9a113fa8123777cb WatchSource:0}: Error finding container dfa29b9759cdc20c876e0dd1c5440c33b674aeb2a327867b9a113fa8123777cb: Status 404 returned error can't find the container with id dfa29b9759cdc20c876e0dd1c5440c33b674aeb2a327867b9a113fa8123777cb
Sep 08 11:51:27.902898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:27.902859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk" event={"ID":"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac","Type":"ContainerStarted","Data":"b856d28673b736131097c937e96d6d870bc1794d48a82824ac492500583d561a"}
Sep 08 11:51:27.903260 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:27.902907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk" event={"ID":"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac","Type":"ContainerStarted","Data":"dfa29b9759cdc20c876e0dd1c5440c33b674aeb2a327867b9a113fa8123777cb"}
Sep 08 11:51:27.923403 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:27.923357    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk" podStartSLOduration=1.923341507 podStartE2EDuration="1.923341507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:51:27.922221223 +0000 UTC m=+1294.488981774" watchObservedRunningTime="2026-09-08 11:51:27.923341507 +0000 UTC m=+1294.490102045"
Sep 08 11:51:32.574747 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:32.574710    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:51:36.944197 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:36.944164    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:36.944722 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:36.944207    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:36.956834 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:36.956801    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:37.965267 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:37.965239    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:51:38.586505 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:38.586448    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:51:54.148006 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:54.147977    2561 scope.go:117] "RemoveContainer" containerID="d5453dc7e1ad3455edcfa1f0f2aab90445f0aa797f055a572d23762f277863c9"
Sep 08 11:51:54.157960 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:54.157922    2561 scope.go:117] "RemoveContainer" containerID="53d0a24763f42b4baf05f260588ef5733d90129a9ed513396676b8a53d73d638"
Sep 08 11:51:54.165396 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:51:54.165375    2561 scope.go:117] "RemoveContainer" containerID="0055c7dd031376fc5468becceef8e9fb00f8f536ccb0a66d0dfb1b8dc21c5ee9"
Sep 08 11:52:09.451120 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:09.451088    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:10.724145 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:10.724112    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:15.879288 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:15.879242    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:32.667063 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:32.667030    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:33.807308 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:33.807266    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"]
Sep 08 11:52:33.832674 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:33.832643    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:35.133000 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:35.132959    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"]
Sep 08 11:52:35.158240 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:35.158206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:41.038217 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:41.038177    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:41.733188 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:41.733154    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:41.832705 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:41.832673    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:42.567093 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:42.567055    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:43.253152 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:43.253113    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:43.355622 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:43.355587    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:43.449553 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:43.449526    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:43.518212 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:43.518142    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:43.619402 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:43.619372    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:44.825832 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:44.825796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:48.826179 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:48.826124    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w" podUID="b03f8b9c-30ac-4e56-b530-2dcac564ed3b" containerName="main" containerID="cri-o://3e51d0fd6c377e189d306958d68bb62c7412f1ac10db12eac515b74ff1713165" gracePeriod=45
Sep 08 11:52:49.071060 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.071035    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:52:49.163593 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.163509    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-tls-certs\") pod \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") "
Sep 08 11:52:49.163593 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.163578    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-model-cache\") pod \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") "
Sep 08 11:52:49.163833 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.163598    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-home\") pod \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") "
Sep 08 11:52:49.163833 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.163640    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czgmv\" (UniqueName: \"kubernetes.io/projected/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-kube-api-access-czgmv\") pod \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") "
Sep 08 11:52:49.163833 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.163680    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-tmp-dir\") pod \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") "
Sep 08 11:52:49.163833 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.163705    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-dshm\") pod \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\" (UID: \"b03f8b9c-30ac-4e56-b530-2dcac564ed3b\") "
Sep 08 11:52:49.163833 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.163785    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-model-cache" (OuterVolumeSpecName: "model-cache") pod "b03f8b9c-30ac-4e56-b530-2dcac564ed3b" (UID: "b03f8b9c-30ac-4e56-b530-2dcac564ed3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:52:49.164085 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.163897    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-home" (OuterVolumeSpecName: "home") pod "b03f8b9c-30ac-4e56-b530-2dcac564ed3b" (UID: "b03f8b9c-30ac-4e56-b530-2dcac564ed3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:52:49.164085 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.163937    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b03f8b9c-30ac-4e56-b530-2dcac564ed3b" (UID: "b03f8b9c-30ac-4e56-b530-2dcac564ed3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:52:49.164085 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.163976    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-model-cache\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:52:49.165654 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.165628    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b03f8b9c-30ac-4e56-b530-2dcac564ed3b" (UID: "b03f8b9c-30ac-4e56-b530-2dcac564ed3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:52:49.165773 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.165757    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-dshm" (OuterVolumeSpecName: "dshm") pod "b03f8b9c-30ac-4e56-b530-2dcac564ed3b" (UID: "b03f8b9c-30ac-4e56-b530-2dcac564ed3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:52:49.165817 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.165786    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-kube-api-access-czgmv" (OuterVolumeSpecName: "kube-api-access-czgmv") pod "b03f8b9c-30ac-4e56-b530-2dcac564ed3b" (UID: "b03f8b9c-30ac-4e56-b530-2dcac564ed3b"). InnerVolumeSpecName "kube-api-access-czgmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:52:49.265015 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.264981    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-tmp-dir\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:52:49.265015 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.265009    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-dshm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:52:49.265015 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.265018    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-tls-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:52:49.265015 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.265027    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-home\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:52:49.265283 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.265036    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czgmv\" (UniqueName: \"kubernetes.io/projected/b03f8b9c-30ac-4e56-b530-2dcac564ed3b-kube-api-access-czgmv\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:52:49.306074 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.306041    2561 generic.go:356] "Generic (PLEG): container finished" podID="b03f8b9c-30ac-4e56-b530-2dcac564ed3b" containerID="3e51d0fd6c377e189d306958d68bb62c7412f1ac10db12eac515b74ff1713165" exitCode=0
Sep 08 11:52:49.306239 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.306083    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"
Sep 08 11:52:49.306239 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.306137    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w" event={"ID":"b03f8b9c-30ac-4e56-b530-2dcac564ed3b","Type":"ContainerDied","Data":"3e51d0fd6c377e189d306958d68bb62c7412f1ac10db12eac515b74ff1713165"}
Sep 08 11:52:49.306239 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.306186    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w" event={"ID":"b03f8b9c-30ac-4e56-b530-2dcac564ed3b","Type":"ContainerDied","Data":"6b61518f028aa5f68eacb305c9f03e3b5082e2ca5ea4e57826b63cfa929d441a"}
Sep 08 11:52:49.306239 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.306208    2561 scope.go:117] "RemoveContainer" containerID="3e51d0fd6c377e189d306958d68bb62c7412f1ac10db12eac515b74ff1713165"
Sep 08 11:52:49.315945 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.315895    2561 scope.go:117] "RemoveContainer" containerID="3e51d0fd6c377e189d306958d68bb62c7412f1ac10db12eac515b74ff1713165"
Sep 08 11:52:49.316402 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:52:49.316375    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e51d0fd6c377e189d306958d68bb62c7412f1ac10db12eac515b74ff1713165\": container with ID starting with 3e51d0fd6c377e189d306958d68bb62c7412f1ac10db12eac515b74ff1713165 not found: ID does not exist" containerID="3e51d0fd6c377e189d306958d68bb62c7412f1ac10db12eac515b74ff1713165"
Sep 08 11:52:49.316517 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.316415    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e51d0fd6c377e189d306958d68bb62c7412f1ac10db12eac515b74ff1713165"} err="failed to get container status \"3e51d0fd6c377e189d306958d68bb62c7412f1ac10db12eac515b74ff1713165\": rpc error: code = NotFound desc = could not find container \"3e51d0fd6c377e189d306958d68bb62c7412f1ac10db12eac515b74ff1713165\": container with ID starting with 3e51d0fd6c377e189d306958d68bb62c7412f1ac10db12eac515b74ff1713165 not found: ID does not exist"
Sep 08 11:52:49.331321 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.331299    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"]
Sep 08 11:52:49.335150 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.335127    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/sub-test-model-838a68-kserve-66bbf854ff-nh75w"]
Sep 08 11:52:49.960928 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:49.960886    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b03f8b9c-30ac-4e56-b530-2dcac564ed3b" path="/var/lib/kubelet/pods/b03f8b9c-30ac-4e56-b530-2dcac564ed3b/volumes"
Sep 08 11:52:50.152283 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.152238    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk" podUID="6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac" containerName="main" containerID="cri-o://b856d28673b736131097c937e96d6d870bc1794d48a82824ac492500583d561a" gracePeriod=45
Sep 08 11:52:50.313528 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.313493    2561 generic.go:356] "Generic (PLEG): container finished" podID="6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac" containerID="b856d28673b736131097c937e96d6d870bc1794d48a82824ac492500583d561a" exitCode=0
Sep 08 11:52:50.313708 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.313557    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk" event={"ID":"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac","Type":"ContainerDied","Data":"b856d28673b736131097c937e96d6d870bc1794d48a82824ac492500583d561a"}
Sep 08 11:52:50.407486 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.407439    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:52:50.475352 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.475318    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-model-cache\") pod \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") "
Sep 08 11:52:50.475576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.475409    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-home\") pod \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") "
Sep 08 11:52:50.475576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.475494    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-dshm\") pod \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") "
Sep 08 11:52:50.475576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.475512    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-tmp-dir\") pod \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") "
Sep 08 11:52:50.475576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.475530    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-tls-certs\") pod \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") "
Sep 08 11:52:50.475576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.475562    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfmwx\" (UniqueName: \"kubernetes.io/projected/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-kube-api-access-jfmwx\") pod \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\" (UID: \"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac\") "
Sep 08 11:52:50.475852 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.475648    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac" (UID: "6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:52:50.475852 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.475700    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-home" (OuterVolumeSpecName: "home") pod "6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac" (UID: "6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:52:50.475852 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.475777    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac" (UID: "6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:52:50.475852 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.475824    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-home\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:52:50.475852 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.475842    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-model-cache\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:52:50.477830 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.477797    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-dshm" (OuterVolumeSpecName: "dshm") pod "6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac" (UID: "6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:52:50.477830 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.477813    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-kube-api-access-jfmwx" (OuterVolumeSpecName: "kube-api-access-jfmwx") pod "6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac" (UID: "6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac"). InnerVolumeSpecName "kube-api-access-jfmwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:52:50.478016 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.477853    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac" (UID: "6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:52:50.577007 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.576922    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-dshm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:52:50.577007 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.576951    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-tmp-dir\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:52:50.577007 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.576961    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-tls-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:52:50.577007 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:50.576971    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfmwx\" (UniqueName: \"kubernetes.io/projected/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac-kube-api-access-jfmwx\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:52:51.157560 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:51.157530    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:51.257436 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:51.257404    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:51.319393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:51.319358    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"
Sep 08 11:52:51.319393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:51.319367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk" event={"ID":"6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac","Type":"ContainerDied","Data":"dfa29b9759cdc20c876e0dd1c5440c33b674aeb2a327867b9a113fa8123777cb"}
Sep 08 11:52:51.319649 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:51.319410    2561 scope.go:117] "RemoveContainer" containerID="b856d28673b736131097c937e96d6d870bc1794d48a82824ac492500583d561a"
Sep 08 11:52:51.343344 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:51.343320    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"]
Sep 08 11:52:51.349849 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:51.349825    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/sub-test-model-bb9b64-kserve-6c86789b85-gqxgk"]
Sep 08 11:52:51.960877 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:51.960843    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac" path="/var/lib/kubelet/pods/6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac/volumes"
Sep 08 11:52:52.569307 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:52.569273    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:54.199650 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:54.199615    2561 scope.go:117] "RemoveContainer" containerID="6790bdc7b6bbcca8d4c783f14d71d479ee417ca1143fff3e8f4947066e8b5134"
Sep 08 11:52:54.207507 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:54.207484    2561 scope.go:117] "RemoveContainer" containerID="bab788a52e68f29f648e70b2e18df7c7c0b22e15f6e08eeb4aa0996655ef5d5d"
Sep 08 11:52:54.214624 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:54.214605    2561 scope.go:117] "RemoveContainer" containerID="568c403a8da3c6682bbef758eb2130b637a3563a69f3158298e0d46364a183a3"
Sep 08 11:52:58.579514 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:58.579460    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:58.684022 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:58.683982    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:58.780768 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:58.780735    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:52:59.080053 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:52:59.080022    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:06.629289 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.629243    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"]
Sep 08 11:53:06.629835 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.629812    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac" containerName="main"
Sep 08 11:53:06.629884 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.629840    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac" containerName="main"
Sep 08 11:53:06.629932 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.629882    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03f8b9c-30ac-4e56-b530-2dcac564ed3b" containerName="main"
Sep 08 11:53:06.629932 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.629892    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03f8b9c-30ac-4e56-b530-2dcac564ed3b" containerName="main"
Sep 08 11:53:06.630014 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.629977    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cfa9f4f-ee2c-461c-9aa8-e0f3dc65b1ac" containerName="main"
Sep 08 11:53:06.630014 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.629989    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03f8b9c-30ac-4e56-b530-2dcac564ed3b" containerName="main"
Sep 08 11:53:06.634674 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.634655    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.638489 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.638452    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bb9b64\"/\"kube-root-ca.crt\""
Sep 08 11:53:06.638710 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.638696    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bb9b64\"/\"openshift-service-ca.crt\""
Sep 08 11:53:06.638764 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.638726    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bb9b64\"/\"default-dockercfg-bbswc\""
Sep 08 11:53:06.638814 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.638726    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bb9b64\"/\"rate-test-model-bb9b64-kserve-self-signed-certs\""
Sep 08 11:53:06.644255 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.644218    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"]
Sep 08 11:53:06.654657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.654635    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:06.814267 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.814226    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sspwc\" (UniqueName: \"kubernetes.io/projected/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-kube-api-access-sspwc\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.814442 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.814275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-tls-certs\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.814442 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.814362    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-home\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.814442 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.814392    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-tmp-dir\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.814442 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.814408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-dshm\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.814611 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.814528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-model-cache\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.915773 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.915684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-model-cache\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.915773 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.915727    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sspwc\" (UniqueName: \"kubernetes.io/projected/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-kube-api-access-sspwc\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.915773 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.915761    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-tls-certs\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.916031 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.915813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-home\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.916031 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.915839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-tmp-dir\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.916031 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.915861    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-dshm\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.916200 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.916179    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-home\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.916245 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.916178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-model-cache\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.916309 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.916251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-tmp-dir\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.918028 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.918007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-dshm\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.918366 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.918349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-tls-certs\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.925129 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.925111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sspwc\" (UniqueName: \"kubernetes.io/projected/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-kube-api-access-sspwc\") pod \"rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:06.946092 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:06.946066    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:07.218535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:07.218509    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"]
Sep 08 11:53:07.220289 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:53:07.220262    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ef8f566_4e2e_49b9_b3f9_25575bd2d32b.slice/crio-5c2383709b4754766653128c6ce416222a14b1062d448d3cb68ae865ebc3e921 WatchSource:0}: Error finding container 5c2383709b4754766653128c6ce416222a14b1062d448d3cb68ae865ebc3e921: Status 404 returned error can't find the container with id 5c2383709b4754766653128c6ce416222a14b1062d448d3cb68ae865ebc3e921
Sep 08 11:53:07.398051 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:07.398019    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7" event={"ID":"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b","Type":"ContainerStarted","Data":"ad6c19367bdc8e2ab7f44408d740738f43d98a8e7ee49ef391ddaae0c336e758"}
Sep 08 11:53:07.398051 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:07.398057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7" event={"ID":"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b","Type":"ContainerStarted","Data":"5c2383709b4754766653128c6ce416222a14b1062d448d3cb68ae865ebc3e921"}
Sep 08 11:53:07.416343 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:07.416293    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7" podStartSLOduration=1.416279903 podStartE2EDuration="1.416279903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:53:07.416015 +0000 UTC m=+1393.982775542" watchObservedRunningTime="2026-09-08 11:53:07.416279903 +0000 UTC m=+1393.983040444"
Sep 08 11:53:09.270615 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:09.270581    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:09.498355 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:09.498323    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:16.947201 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:16.947156    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:16.947671 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:16.947346    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:16.959621 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:16.959597    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:17.457306 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:17.457272    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:53:30.425411 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:30.425373    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:36.925138 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:36.925096    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:37.025229 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:37.025169    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:37.277708 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:37.277675    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:37.600527 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:37.600429    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:37.777030 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:37.776998    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:40.465412 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:40.465380    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:40.587067 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:40.587028    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:41.642196 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:41.642162    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:41.742757 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:41.742726    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:41.842377 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:41.842350    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:42.284560 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.284518    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:42.380482 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.380435    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:42.421446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.421415    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"]
Sep 08 11:53:42.424956 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.424941    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"
Sep 08 11:53:42.427979 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.427960    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-77fac0\""
Sep 08 11:53:42.428095 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.428000    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-77fac0-dockercfg-n5fwk\""
Sep 08 11:53:42.436407 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.436382    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"]
Sep 08 11:53:42.480128 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.480105    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"]
Sep 08 11:53:42.483358 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.483341    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"
Sep 08 11:53:42.493683 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.493663    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"]
Sep 08 11:53:42.512737 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.512711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d2ec305-9152-43a2-b3c3-56ad9e271632-plugins-config-volume\") pod \"payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2\" (UID: \"5d2ec305-9152-43a2-b3c3-56ad9e271632\") " pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"
Sep 08 11:53:42.512959 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.512937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4gd\" (UniqueName: \"kubernetes.io/projected/5d2ec305-9152-43a2-b3c3-56ad9e271632-kube-api-access-gb4gd\") pod \"payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2\" (UID: \"5d2ec305-9152-43a2-b3c3-56ad9e271632\") " pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"
Sep 08 11:53:42.614280 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.614200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d2ec305-9152-43a2-b3c3-56ad9e271632-plugins-config-volume\") pod \"payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2\" (UID: \"5d2ec305-9152-43a2-b3c3-56ad9e271632\") " pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"
Sep 08 11:53:42.614427 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.614352    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4gd\" (UniqueName: \"kubernetes.io/projected/5d2ec305-9152-43a2-b3c3-56ad9e271632-kube-api-access-gb4gd\") pod \"payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2\" (UID: \"5d2ec305-9152-43a2-b3c3-56ad9e271632\") " pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"
Sep 08 11:53:42.614427 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.614405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c611be-bb28-460f-951d-83bb422f39da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh\" (UID: \"26c611be-bb28-460f-951d-83bb422f39da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"
Sep 08 11:53:42.614596 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.614482    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhzw4\" (UniqueName: \"kubernetes.io/projected/26c611be-bb28-460f-951d-83bb422f39da-kube-api-access-rhzw4\") pod \"payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh\" (UID: \"26c611be-bb28-460f-951d-83bb422f39da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"
Sep 08 11:53:42.614975 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.614954    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d2ec305-9152-43a2-b3c3-56ad9e271632-plugins-config-volume\") pod \"payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2\" (UID: \"5d2ec305-9152-43a2-b3c3-56ad9e271632\") " pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"
Sep 08 11:53:42.625005 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.624985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4gd\" (UniqueName: \"kubernetes.io/projected/5d2ec305-9152-43a2-b3c3-56ad9e271632-kube-api-access-gb4gd\") pod \"payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2\" (UID: \"5d2ec305-9152-43a2-b3c3-56ad9e271632\") " pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"
Sep 08 11:53:42.715893 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.715864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c611be-bb28-460f-951d-83bb422f39da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh\" (UID: \"26c611be-bb28-460f-951d-83bb422f39da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"
Sep 08 11:53:42.716274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.715901    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhzw4\" (UniqueName: \"kubernetes.io/projected/26c611be-bb28-460f-951d-83bb422f39da-kube-api-access-rhzw4\") pod \"payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh\" (UID: \"26c611be-bb28-460f-951d-83bb422f39da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"
Sep 08 11:53:42.716558 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.716539    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c611be-bb28-460f-951d-83bb422f39da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh\" (UID: \"26c611be-bb28-460f-951d-83bb422f39da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"
Sep 08 11:53:42.725535 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.725512    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhzw4\" (UniqueName: \"kubernetes.io/projected/26c611be-bb28-460f-951d-83bb422f39da-kube-api-access-rhzw4\") pod \"payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh\" (UID: \"26c611be-bb28-460f-951d-83bb422f39da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"
Sep 08 11:53:42.735447 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.735426    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"
Sep 08 11:53:42.793716 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:42.793459    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"
Sep 08 11:53:43.013451 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:43.013423    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"]
Sep 08 11:53:43.015969 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:53:43.015945    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2ec305_9152_43a2_b3c3_56ad9e271632.slice/crio-af64befc3e71e0a3edd637c59d39b0794327869c6c44ae2a081c79e864dace5f WatchSource:0}: Error finding container af64befc3e71e0a3edd637c59d39b0794327869c6c44ae2a081c79e864dace5f: Status 404 returned error can't find the container with id af64befc3e71e0a3edd637c59d39b0794327869c6c44ae2a081c79e864dace5f
Sep 08 11:53:43.072696 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:43.072664    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"]
Sep 08 11:53:43.074879 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:53:43.074853    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c611be_bb28_460f_951d_83bb422f39da.slice/crio-5a873d140826af4e6711dc9a621aa8e39541ce47a8ba25c3061890806e5b72e2 WatchSource:0}: Error finding container 5a873d140826af4e6711dc9a621aa8e39541ce47a8ba25c3061890806e5b72e2: Status 404 returned error can't find the container with id 5a873d140826af4e6711dc9a621aa8e39541ce47a8ba25c3061890806e5b72e2
Sep 08 11:53:43.569661 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:43.569601    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh" event={"ID":"26c611be-bb28-460f-951d-83bb422f39da","Type":"ContainerStarted","Data":"60edb3d19c94fac21a6d2543a2a3bea60510f9f6fc3ddef46224b345af57f06b"}
Sep 08 11:53:43.569835 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:43.569706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh" event={"ID":"26c611be-bb28-460f-951d-83bb422f39da","Type":"ContainerStarted","Data":"5a873d140826af4e6711dc9a621aa8e39541ce47a8ba25c3061890806e5b72e2"}
Sep 08 11:53:43.569835 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:43.569738    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"
Sep 08 11:53:43.571700 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:43.571672    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2" event={"ID":"5d2ec305-9152-43a2-b3c3-56ad9e271632","Type":"ContainerStarted","Data":"57611255b96323ca08f47e41236a7b20516afde0738cb61f815b9436219dcc17"}
Sep 08 11:53:43.571797 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:43.571720    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2" event={"ID":"5d2ec305-9152-43a2-b3c3-56ad9e271632","Type":"ContainerStarted","Data":"af64befc3e71e0a3edd637c59d39b0794327869c6c44ae2a081c79e864dace5f"}
Sep 08 11:53:43.571797 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:43.571745    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"
Sep 08 11:53:43.587185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:43.587145    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh" podStartSLOduration=1.5871320770000001 podStartE2EDuration="1.587132077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:53:43.586203368 +0000 UTC m=+1430.152963919" watchObservedRunningTime="2026-09-08 11:53:43.587132077 +0000 UTC m=+1430.153892618"
Sep 08 11:53:43.604476 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:43.604408    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2" podStartSLOduration=1.604393517 podStartE2EDuration="1.604393517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:53:43.603128123 +0000 UTC m=+1430.169888664" watchObservedRunningTime="2026-09-08 11:53:43.604393517 +0000 UTC m=+1430.171154058"
Sep 08 11:53:50.339244 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:50.339153    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:51.223495 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:51.223444    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"]
Sep 08 11:53:51.247868 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:51.247837    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:51.697935 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:51.697858    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:52.012000 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:52.011965    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:52.138427 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:52.138389    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:52.365977 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:52.365890    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:54.248130 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:54.248096    2561 scope.go:117] "RemoveContainer" containerID="f425ee29a67d3088bc9063de9f60378adafb304d530f0719780a69e85f70cf8d"
Sep 08 11:53:54.579557 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:54.579456    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"
Sep 08 11:53:54.579691 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:54.579563    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"
Sep 08 11:53:55.004853 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:55.004795    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:55.099821 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:55.099776    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:56.354645 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:56.354606    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:56.451671 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:56.451638    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:56.973792 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:56.973753    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"]
Sep 08 11:53:56.978491 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:56.978455    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"
Sep 08 11:53:56.981039 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:56.981018    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3f9288\""
Sep 08 11:53:56.981147 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:56.981023    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3f9288-dockercfg-rqhbk\""
Sep 08 11:53:56.988190 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:56.988165    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"]
Sep 08 11:53:57.138483 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.138443    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad0d1b7-297e-4099-8651-4c3683684140-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8\" (UID: \"0ad0d1b7-297e-4099-8651-4c3683684140\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"
Sep 08 11:53:57.138646 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.138559    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntjpr\" (UniqueName: \"kubernetes.io/projected/0ad0d1b7-297e-4099-8651-4c3683684140-kube-api-access-ntjpr\") pod \"payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8\" (UID: \"0ad0d1b7-297e-4099-8651-4c3683684140\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"
Sep 08 11:53:57.239964 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.239879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad0d1b7-297e-4099-8651-4c3683684140-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8\" (UID: \"0ad0d1b7-297e-4099-8651-4c3683684140\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"
Sep 08 11:53:57.240119 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.240045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntjpr\" (UniqueName: \"kubernetes.io/projected/0ad0d1b7-297e-4099-8651-4c3683684140-kube-api-access-ntjpr\") pod \"payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8\" (UID: \"0ad0d1b7-297e-4099-8651-4c3683684140\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"
Sep 08 11:53:57.240654 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.240636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad0d1b7-297e-4099-8651-4c3683684140-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8\" (UID: \"0ad0d1b7-297e-4099-8651-4c3683684140\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"
Sep 08 11:53:57.248701 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.248682    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntjpr\" (UniqueName: \"kubernetes.io/projected/0ad0d1b7-297e-4099-8651-4c3683684140-kube-api-access-ntjpr\") pod \"payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8\" (UID: \"0ad0d1b7-297e-4099-8651-4c3683684140\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"
Sep 08 11:53:57.289791 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.289764    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"
Sep 08 11:53:57.560880 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.560839    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"]
Sep 08 11:53:57.563233 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:53:57.563205    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad0d1b7_297e_4099_8651_4c3683684140.slice/crio-7654cff4644d1de4e5394c5fabd6d0f76784bf61e82f33a67571910f1184afd9 WatchSource:0}: Error finding container 7654cff4644d1de4e5394c5fabd6d0f76784bf61e82f33a67571910f1184afd9: Status 404 returned error can't find the container with id 7654cff4644d1de4e5394c5fabd6d0f76784bf61e82f33a67571910f1184afd9
Sep 08 11:53:57.593505 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.593450    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:53:57.644783 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.644755    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8" event={"ID":"0ad0d1b7-297e-4099-8651-4c3683684140","Type":"ContainerStarted","Data":"f32f642f0f18008633ebe41cc533def35c3f137581e39b0e1c9d8c29f1605c95"}
Sep 08 11:53:57.644905 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.644813    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8" event={"ID":"0ad0d1b7-297e-4099-8651-4c3683684140","Type":"ContainerStarted","Data":"7654cff4644d1de4e5394c5fabd6d0f76784bf61e82f33a67571910f1184afd9"}
Sep 08 11:53:57.644905 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.644896    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"
Sep 08 11:53:57.664427 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.664373    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8" podStartSLOduration=1.664353337 podStartE2EDuration="1.664353337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:53:57.66272768 +0000 UTC m=+1444.229488223" watchObservedRunningTime="2026-09-08 11:53:57.664353337 +0000 UTC m=+1444.231113879"
Sep 08 11:53:57.671308 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:53:57.671274    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:02.819570 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:02.819536    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:05.473519 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:05.473439    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:05.918369 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:05.918273    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:06.242878 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.242837    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:06.243185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.243154    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7" podUID="0ef8f566-4e2e-49b9-b3f9-25575bd2d32b" containerName="main" containerID="cri-o://ad6c19367bdc8e2ab7f44408d740738f43d98a8e7ee49ef391ddaae0c336e758" gracePeriod=45
Sep 08 11:54:06.494083 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.494015    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:54:06.622868 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.622835    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-model-cache\") pod \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") "
Sep 08 11:54:06.622868 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.622872    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sspwc\" (UniqueName: \"kubernetes.io/projected/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-kube-api-access-sspwc\") pod \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") "
Sep 08 11:54:06.623114 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.622919    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-tls-certs\") pod \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") "
Sep 08 11:54:06.623114 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.622946    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-home\") pod \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") "
Sep 08 11:54:06.623114 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.622960    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-tmp-dir\") pod \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") "
Sep 08 11:54:06.623114 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.622983    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-dshm\") pod \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\" (UID: \"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b\") "
Sep 08 11:54:06.623338 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.623141    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ef8f566-4e2e-49b9-b3f9-25575bd2d32b" (UID: "0ef8f566-4e2e-49b9-b3f9-25575bd2d32b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:54:06.623338 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.623271    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-home" (OuterVolumeSpecName: "home") pod "0ef8f566-4e2e-49b9-b3f9-25575bd2d32b" (UID: "0ef8f566-4e2e-49b9-b3f9-25575bd2d32b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:54:06.623338 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.623288    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-model-cache\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:54:06.623338 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.623285    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ef8f566-4e2e-49b9-b3f9-25575bd2d32b" (UID: "0ef8f566-4e2e-49b9-b3f9-25575bd2d32b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:54:06.625186 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.625153    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-dshm" (OuterVolumeSpecName: "dshm") pod "0ef8f566-4e2e-49b9-b3f9-25575bd2d32b" (UID: "0ef8f566-4e2e-49b9-b3f9-25575bd2d32b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:54:06.625302 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.625199    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ef8f566-4e2e-49b9-b3f9-25575bd2d32b" (UID: "0ef8f566-4e2e-49b9-b3f9-25575bd2d32b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:54:06.625302 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.625230    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-kube-api-access-sspwc" (OuterVolumeSpecName: "kube-api-access-sspwc") pod "0ef8f566-4e2e-49b9-b3f9-25575bd2d32b" (UID: "0ef8f566-4e2e-49b9-b3f9-25575bd2d32b"). InnerVolumeSpecName "kube-api-access-sspwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:54:06.690779 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.690747    2561 generic.go:356] "Generic (PLEG): container finished" podID="0ef8f566-4e2e-49b9-b3f9-25575bd2d32b" containerID="ad6c19367bdc8e2ab7f44408d740738f43d98a8e7ee49ef391ddaae0c336e758" exitCode=0
Sep 08 11:54:06.690936 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.690804    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"
Sep 08 11:54:06.690936 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.690848    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7" event={"ID":"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b","Type":"ContainerDied","Data":"ad6c19367bdc8e2ab7f44408d740738f43d98a8e7ee49ef391ddaae0c336e758"}
Sep 08 11:54:06.690936 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.690896    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7" event={"ID":"0ef8f566-4e2e-49b9-b3f9-25575bd2d32b","Type":"ContainerDied","Data":"5c2383709b4754766653128c6ce416222a14b1062d448d3cb68ae865ebc3e921"}
Sep 08 11:54:06.690936 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.690919    2561 scope.go:117] "RemoveContainer" containerID="ad6c19367bdc8e2ab7f44408d740738f43d98a8e7ee49ef391ddaae0c336e758"
Sep 08 11:54:06.700323 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.700267    2561 scope.go:117] "RemoveContainer" containerID="ad6c19367bdc8e2ab7f44408d740738f43d98a8e7ee49ef391ddaae0c336e758"
Sep 08 11:54:06.700647 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:54:06.700586    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6c19367bdc8e2ab7f44408d740738f43d98a8e7ee49ef391ddaae0c336e758\": container with ID starting with ad6c19367bdc8e2ab7f44408d740738f43d98a8e7ee49ef391ddaae0c336e758 not found: ID does not exist" containerID="ad6c19367bdc8e2ab7f44408d740738f43d98a8e7ee49ef391ddaae0c336e758"
Sep 08 11:54:06.700647 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.700621    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6c19367bdc8e2ab7f44408d740738f43d98a8e7ee49ef391ddaae0c336e758"} err="failed to get container status \"ad6c19367bdc8e2ab7f44408d740738f43d98a8e7ee49ef391ddaae0c336e758\": rpc error: code = NotFound desc = could not find container \"ad6c19367bdc8e2ab7f44408d740738f43d98a8e7ee49ef391ddaae0c336e758\": container with ID starting with ad6c19367bdc8e2ab7f44408d740738f43d98a8e7ee49ef391ddaae0c336e758 not found: ID does not exist"
Sep 08 11:54:06.724641 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.724623    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-dshm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:54:06.724641 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.724641    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sspwc\" (UniqueName: \"kubernetes.io/projected/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-kube-api-access-sspwc\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:54:06.724769 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.724650    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-tls-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:54:06.724769 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.724660    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-home\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:54:06.724769 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.724669    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b-tmp-dir\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:54:06.734070 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.734049    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"]
Sep 08 11:54:06.741356 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.741337    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/rate-test-model-bb9b64-kserve-86f8bd4fc5-sjcn7"]
Sep 08 11:54:06.755407 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:06.755349    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:07.961999 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:07.961966    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ef8f566-4e2e-49b9-b3f9-25575bd2d32b" path="/var/lib/kubelet/pods/0ef8f566-4e2e-49b9-b3f9-25575bd2d32b/volumes"
Sep 08 11:54:08.653285 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:08.653253    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"
Sep 08 11:54:09.703681 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:09.703647    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:09.798741 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:09.798702    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:10.861305 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:10.861270    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:10.926895 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:10.926861    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:11.031204 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.031168    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:11.625932 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.625894    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"]
Sep 08 11:54:11.626339 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.626321    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ef8f566-4e2e-49b9-b3f9-25575bd2d32b" containerName="main"
Sep 08 11:54:11.626441 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.626342    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ef8f566-4e2e-49b9-b3f9-25575bd2d32b" containerName="main"
Sep 08 11:54:11.626441 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.626436    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ef8f566-4e2e-49b9-b3f9-25575bd2d32b" containerName="main"
Sep 08 11:54:11.630454 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.630434    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"
Sep 08 11:54:11.632542 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.632520    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"]
Sep 08 11:54:11.633277 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.633252    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ca8280-dockercfg-6b2r4\""
Sep 08 11:54:11.633501 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.633483    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ca8280\""
Sep 08 11:54:11.635761 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.635743    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"
Sep 08 11:54:11.641505 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.641483    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"]
Sep 08 11:54:11.653266 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.650649    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"]
Sep 08 11:54:11.664552 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.664526    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56749441-a582-403b-95c8-fb680a7eb178-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc\" (UID: \"56749441-a582-403b-95c8-fb680a7eb178\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"
Sep 08 11:54:11.664697 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.664568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5mr\" (UniqueName: \"kubernetes.io/projected/56749441-a582-403b-95c8-fb680a7eb178-kube-api-access-cb5mr\") pod \"payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc\" (UID: \"56749441-a582-403b-95c8-fb680a7eb178\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"
Sep 08 11:54:11.664697 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.664631    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5dfc\" (UniqueName: \"kubernetes.io/projected/0c327595-0cae-4ee3-adaa-9b3db6059cbb-kube-api-access-t5dfc\") pod \"payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99\" (UID: \"0c327595-0cae-4ee3-adaa-9b3db6059cbb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"
Sep 08 11:54:11.664697 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.664662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c327595-0cae-4ee3-adaa-9b3db6059cbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99\" (UID: \"0c327595-0cae-4ee3-adaa-9b3db6059cbb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"
Sep 08 11:54:11.765961 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.765927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56749441-a582-403b-95c8-fb680a7eb178-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc\" (UID: \"56749441-a582-403b-95c8-fb680a7eb178\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"
Sep 08 11:54:11.765961 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.765966    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5mr\" (UniqueName: \"kubernetes.io/projected/56749441-a582-403b-95c8-fb680a7eb178-kube-api-access-cb5mr\") pod \"payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc\" (UID: \"56749441-a582-403b-95c8-fb680a7eb178\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"
Sep 08 11:54:11.766173 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.766008    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5dfc\" (UniqueName: \"kubernetes.io/projected/0c327595-0cae-4ee3-adaa-9b3db6059cbb-kube-api-access-t5dfc\") pod \"payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99\" (UID: \"0c327595-0cae-4ee3-adaa-9b3db6059cbb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"
Sep 08 11:54:11.766173 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.766036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c327595-0cae-4ee3-adaa-9b3db6059cbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99\" (UID: \"0c327595-0cae-4ee3-adaa-9b3db6059cbb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"
Sep 08 11:54:11.766703 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.766681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c327595-0cae-4ee3-adaa-9b3db6059cbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99\" (UID: \"0c327595-0cae-4ee3-adaa-9b3db6059cbb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"
Sep 08 11:54:11.766750 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.766691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56749441-a582-403b-95c8-fb680a7eb178-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc\" (UID: \"56749441-a582-403b-95c8-fb680a7eb178\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"
Sep 08 11:54:11.775113 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.775082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5mr\" (UniqueName: \"kubernetes.io/projected/56749441-a582-403b-95c8-fb680a7eb178-kube-api-access-cb5mr\") pod \"payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc\" (UID: \"56749441-a582-403b-95c8-fb680a7eb178\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"
Sep 08 11:54:11.775238 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.775112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5dfc\" (UniqueName: \"kubernetes.io/projected/0c327595-0cae-4ee3-adaa-9b3db6059cbb-kube-api-access-t5dfc\") pod \"payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99\" (UID: \"0c327595-0cae-4ee3-adaa-9b3db6059cbb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"
Sep 08 11:54:11.943253 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.943162    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"
Sep 08 11:54:11.959032 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:11.959006    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"
Sep 08 11:54:12.220723 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:12.220673    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"]
Sep 08 11:54:12.222767 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:54:12.222740    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c327595_0cae_4ee3_adaa_9b3db6059cbb.slice/crio-a354af7a27b857d64e4b7034d6beb3046d67682011190cd5cb673f97b8843ea0 WatchSource:0}: Error finding container a354af7a27b857d64e4b7034d6beb3046d67682011190cd5cb673f97b8843ea0: Status 404 returned error can't find the container with id a354af7a27b857d64e4b7034d6beb3046d67682011190cd5cb673f97b8843ea0
Sep 08 11:54:12.239415 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:12.239391    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"]
Sep 08 11:54:12.242186 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:54:12.242163    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56749441_a582_403b_95c8_fb680a7eb178.slice/crio-5938af7dcad1bc61b64b5cbf34c827c9d9c3428d8eab7c5727de89d241d3054d WatchSource:0}: Error finding container 5938af7dcad1bc61b64b5cbf34c827c9d9c3428d8eab7c5727de89d241d3054d: Status 404 returned error can't find the container with id 5938af7dcad1bc61b64b5cbf34c827c9d9c3428d8eab7c5727de89d241d3054d
Sep 08 11:54:12.723408 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:12.723373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99" event={"ID":"0c327595-0cae-4ee3-adaa-9b3db6059cbb","Type":"ContainerStarted","Data":"daf7fe79137c71b0c7267d34c921645ac19a55eb6ed5acd57f3f7f630bbd4924"}
Sep 08 11:54:12.723628 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:12.723433    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99" event={"ID":"0c327595-0cae-4ee3-adaa-9b3db6059cbb","Type":"ContainerStarted","Data":"a354af7a27b857d64e4b7034d6beb3046d67682011190cd5cb673f97b8843ea0"}
Sep 08 11:54:12.723628 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:12.723457    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"
Sep 08 11:54:12.725064 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:12.725041    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc" event={"ID":"56749441-a582-403b-95c8-fb680a7eb178","Type":"ContainerStarted","Data":"39985ab27473220b355def23de94a9e583c08705c00a8433647059dd0969f2ab"}
Sep 08 11:54:12.725175 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:12.725078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc" event={"ID":"56749441-a582-403b-95c8-fb680a7eb178","Type":"ContainerStarted","Data":"5938af7dcad1bc61b64b5cbf34c827c9d9c3428d8eab7c5727de89d241d3054d"}
Sep 08 11:54:12.725230 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:12.725169    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"
Sep 08 11:54:12.740327 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:12.740279    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99" podStartSLOduration=1.7402633779999999 podStartE2EDuration="1.740263378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:54:12.740093704 +0000 UTC m=+1459.306854247" watchObservedRunningTime="2026-09-08 11:54:12.740263378 +0000 UTC m=+1459.307023914"
Sep 08 11:54:12.759190 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:12.759137    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc" podStartSLOduration=1.7591195800000001 podStartE2EDuration="1.75911958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:54:12.757388251 +0000 UTC m=+1459.324148796" watchObservedRunningTime="2026-09-08 11:54:12.75911958 +0000 UTC m=+1459.325880127"
Sep 08 11:54:12.970780 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:12.970750    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:13.071702 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:13.071586    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:17.275527 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:17.275492    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:19.985182 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:19.985137    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:20.674003 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:20.673948    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:20.984191 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:20.984148    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:21.263242 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:21.263158    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:21.496819 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:21.496786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:23.731636 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:23.731604    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"
Sep 08 11:54:23.732014 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:23.731675    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"
Sep 08 11:54:24.195002 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:24.194913    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:24.293666 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:24.293627    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:25.450923 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:25.450891    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:25.550845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:25.550814    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:25.652440 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:25.652403    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:26.002390 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.002360    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:26.104141 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.104106    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:26.157939 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.157905    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"]
Sep 08 11:54:26.161392 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.161378    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"
Sep 08 11:54:26.164332 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.164310    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2ebeba\""
Sep 08 11:54:26.164495 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.164461    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2ebeba-dockercfg-qw56b\""
Sep 08 11:54:26.178140 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.178111    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"]
Sep 08 11:54:26.285618 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.285534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56116325-83f2-4cbf-86d7-3348157096a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r\" (UID: \"56116325-83f2-4cbf-86d7-3348157096a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"
Sep 08 11:54:26.285759 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.285648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nfgw\" (UniqueName: \"kubernetes.io/projected/56116325-83f2-4cbf-86d7-3348157096a0-kube-api-access-8nfgw\") pod \"payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r\" (UID: \"56116325-83f2-4cbf-86d7-3348157096a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"
Sep 08 11:54:26.386781 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.386749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nfgw\" (UniqueName: \"kubernetes.io/projected/56116325-83f2-4cbf-86d7-3348157096a0-kube-api-access-8nfgw\") pod \"payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r\" (UID: \"56116325-83f2-4cbf-86d7-3348157096a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"
Sep 08 11:54:26.386944 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.386811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56116325-83f2-4cbf-86d7-3348157096a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r\" (UID: \"56116325-83f2-4cbf-86d7-3348157096a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"
Sep 08 11:54:26.387453 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.387436    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56116325-83f2-4cbf-86d7-3348157096a0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r\" (UID: \"56116325-83f2-4cbf-86d7-3348157096a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"
Sep 08 11:54:26.396268 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.396239    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nfgw\" (UniqueName: \"kubernetes.io/projected/56116325-83f2-4cbf-86d7-3348157096a0-kube-api-access-8nfgw\") pod \"payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r\" (UID: \"56116325-83f2-4cbf-86d7-3348157096a0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"
Sep 08 11:54:26.472067 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.472034    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"
Sep 08 11:54:26.750857 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.750831    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"]
Sep 08 11:54:26.753008 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:54:26.752971    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56116325_83f2_4cbf_86d7_3348157096a0.slice/crio-6e91260a0d17e513c398f11b4cb0072d235d1e4ffac5446fc46fdc37972ec9cc WatchSource:0}: Error finding container 6e91260a0d17e513c398f11b4cb0072d235d1e4ffac5446fc46fdc37972ec9cc: Status 404 returned error can't find the container with id 6e91260a0d17e513c398f11b4cb0072d235d1e4ffac5446fc46fdc37972ec9cc
Sep 08 11:54:26.798235 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:26.798212    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r" event={"ID":"56116325-83f2-4cbf-86d7-3348157096a0","Type":"ContainerStarted","Data":"6e91260a0d17e513c398f11b4cb0072d235d1e4ffac5446fc46fdc37972ec9cc"}
Sep 08 11:54:27.804512 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:27.804455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r" event={"ID":"56116325-83f2-4cbf-86d7-3348157096a0","Type":"ContainerStarted","Data":"75d3a419283e00250ed88c1ce7d484dea2b7dae85f9b5e78ced47f923a33747d"}
Sep 08 11:54:27.804896 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:27.804576    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"
Sep 08 11:54:27.823151 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:27.823092    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r" podStartSLOduration=1.823072904 podStartE2EDuration="1.823072904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:54:27.822254826 +0000 UTC m=+1474.389015367" watchObservedRunningTime="2026-09-08 11:54:27.823072904 +0000 UTC m=+1474.389833446"
Sep 08 11:54:32.076490 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:32.075532    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:34.660490 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:34.660439    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:37.536579 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:37.536537    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w"]
Sep 08 11:54:37.540067 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:37.540041    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w"
Sep 08 11:54:37.542247 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:37.542223    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w"]
Sep 08 11:54:37.543123 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:37.542928    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:54:37.543247 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:37.543151    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:54:37.654850 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:37.654812    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:37.692083 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:37.692050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3451748c-06cc-426b-81a7-a33cfe7b4dc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77fac0-q276w\" (UID: \"3451748c-06cc-426b-81a7-a33cfe7b4dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w"
Sep 08 11:54:37.793485 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:37.793395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3451748c-06cc-426b-81a7-a33cfe7b4dc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77fac0-q276w\" (UID: \"3451748c-06cc-426b-81a7-a33cfe7b4dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w"
Sep 08 11:54:37.794035 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:37.794014    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3451748c-06cc-426b-81a7-a33cfe7b4dc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77fac0-q276w\" (UID: \"3451748c-06cc-426b-81a7-a33cfe7b4dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w"
Sep 08 11:54:37.853713 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:37.853685    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w"
Sep 08 11:54:38.132531 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:38.132506    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w"]
Sep 08 11:54:38.134630 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:54:38.134606    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3451748c_06cc_426b_81a7_a33cfe7b4dc9.slice/crio-623d77221d133e4254f43f0b891c92bcce851985ea02720e68d8ca400a627a18 WatchSource:0}: Error finding container 623d77221d133e4254f43f0b891c92bcce851985ea02720e68d8ca400a627a18: Status 404 returned error can't find the container with id 623d77221d133e4254f43f0b891c92bcce851985ea02720e68d8ca400a627a18
Sep 08 11:54:38.812143 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:38.812117    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"
Sep 08 11:54:38.866321 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:38.866157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w" event={"ID":"3451748c-06cc-426b-81a7-a33cfe7b4dc9","Type":"ContainerStarted","Data":"433c9758343e30749b8d73b98fa649a13253dc90b21549076fcb5e4fca8b922d"}
Sep 08 11:54:38.866321 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:38.866220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w" event={"ID":"3451748c-06cc-426b-81a7-a33cfe7b4dc9","Type":"ContainerStarted","Data":"623d77221d133e4254f43f0b891c92bcce851985ea02720e68d8ca400a627a18"}
Sep 08 11:54:38.886094 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:38.886036    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w" podStartSLOduration=1.886015905 podStartE2EDuration="1.886015905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:54:38.884791909 +0000 UTC m=+1485.451552450" watchObservedRunningTime="2026-09-08 11:54:38.886015905 +0000 UTC m=+1485.452776447"
Sep 08 11:54:39.872493 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:39.872436    2561 generic.go:356] "Generic (PLEG): container finished" podID="3451748c-06cc-426b-81a7-a33cfe7b4dc9" containerID="433c9758343e30749b8d73b98fa649a13253dc90b21549076fcb5e4fca8b922d" exitCode=0
Sep 08 11:54:39.872890 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:39.872524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w" event={"ID":"3451748c-06cc-426b-81a7-a33cfe7b4dc9","Type":"ContainerDied","Data":"433c9758343e30749b8d73b98fa649a13253dc90b21549076fcb5e4fca8b922d"}
Sep 08 11:54:41.010897 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:41.010871    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w"
Sep 08 11:54:41.124821 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:41.124792    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3451748c-06cc-426b-81a7-a33cfe7b4dc9-maas-api-service-ca\") pod \"3451748c-06cc-426b-81a7-a33cfe7b4dc9\" (UID: \"3451748c-06cc-426b-81a7-a33cfe7b4dc9\") "
Sep 08 11:54:41.125176 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:41.125154    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3451748c-06cc-426b-81a7-a33cfe7b4dc9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3451748c-06cc-426b-81a7-a33cfe7b4dc9" (UID: "3451748c-06cc-426b-81a7-a33cfe7b4dc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:54:41.226190 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:41.226163    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3451748c-06cc-426b-81a7-a33cfe7b4dc9-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:54:41.883662 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:41.883624    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w" event={"ID":"3451748c-06cc-426b-81a7-a33cfe7b4dc9","Type":"ContainerDied","Data":"623d77221d133e4254f43f0b891c92bcce851985ea02720e68d8ca400a627a18"}
Sep 08 11:54:41.883809 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:41.883699    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="623d77221d133e4254f43f0b891c92bcce851985ea02720e68d8ca400a627a18"
Sep 08 11:54:41.883809 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:41.883634    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w"
Sep 08 11:54:46.469178 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:46.469146    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:49.265788 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:49.265749    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:52.704684 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.704656    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:52.739255 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.739219    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"]
Sep 08 11:54:52.739542 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.739482    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2" podUID="5d2ec305-9152-43a2-b3c3-56ad9e271632" containerName="payload-processing" containerID="cri-o://57611255b96323ca08f47e41236a7b20516afde0738cb61f815b9436219dcc17" gracePeriod=30
Sep 08 11:54:52.745829 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.745801    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"]
Sep 08 11:54:52.746035 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.746009    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh" podUID="26c611be-bb28-460f-951d-83bb422f39da" containerName="payload-pre-processing" containerID="cri-o://60edb3d19c94fac21a6d2543a2a3bea60510f9f6fc3ddef46224b345af57f06b" gracePeriod=30
Sep 08 11:54:52.832631 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.832603    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:52.926585 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.926551    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w"]
Sep 08 11:54:52.929401 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.929368    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77fac0-q276w"]
Sep 08 11:54:52.939576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.939546    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:52.939805 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.939774    2561 generic.go:356] "Generic (PLEG): container finished" podID="26c611be-bb28-460f-951d-83bb422f39da" containerID="60edb3d19c94fac21a6d2543a2a3bea60510f9f6fc3ddef46224b345af57f06b" exitCode=0
Sep 08 11:54:52.939885 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.939846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh" event={"ID":"26c611be-bb28-460f-951d-83bb422f39da","Type":"ContainerDied","Data":"60edb3d19c94fac21a6d2543a2a3bea60510f9f6fc3ddef46224b345af57f06b"}
Sep 08 11:54:52.941603 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.941579    2561 generic.go:356] "Generic (PLEG): container finished" podID="5d2ec305-9152-43a2-b3c3-56ad9e271632" containerID="57611255b96323ca08f47e41236a7b20516afde0738cb61f815b9436219dcc17" exitCode=0
Sep 08 11:54:52.941729 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.941611    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2" event={"ID":"5d2ec305-9152-43a2-b3c3-56ad9e271632","Type":"ContainerDied","Data":"57611255b96323ca08f47e41236a7b20516afde0738cb61f815b9436219dcc17"}
Sep 08 11:54:52.993182 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:52.993150    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"
Sep 08 11:54:53.024308 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.024284    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"
Sep 08 11:54:53.133505 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.133457    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d2ec305-9152-43a2-b3c3-56ad9e271632-plugins-config-volume\") pod \"5d2ec305-9152-43a2-b3c3-56ad9e271632\" (UID: \"5d2ec305-9152-43a2-b3c3-56ad9e271632\") "
Sep 08 11:54:53.133686 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.133516    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb4gd\" (UniqueName: \"kubernetes.io/projected/5d2ec305-9152-43a2-b3c3-56ad9e271632-kube-api-access-gb4gd\") pod \"5d2ec305-9152-43a2-b3c3-56ad9e271632\" (UID: \"5d2ec305-9152-43a2-b3c3-56ad9e271632\") "
Sep 08 11:54:53.133686 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.133604    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c611be-bb28-460f-951d-83bb422f39da-plugins-config-volume\") pod \"26c611be-bb28-460f-951d-83bb422f39da\" (UID: \"26c611be-bb28-460f-951d-83bb422f39da\") "
Sep 08 11:54:53.133686 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.133649    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhzw4\" (UniqueName: \"kubernetes.io/projected/26c611be-bb28-460f-951d-83bb422f39da-kube-api-access-rhzw4\") pod \"26c611be-bb28-460f-951d-83bb422f39da\" (UID: \"26c611be-bb28-460f-951d-83bb422f39da\") "
Sep 08 11:54:53.133942 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.133916    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2ec305-9152-43a2-b3c3-56ad9e271632-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d2ec305-9152-43a2-b3c3-56ad9e271632" (UID: "5d2ec305-9152-43a2-b3c3-56ad9e271632"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:54:53.134018 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.133982    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c611be-bb28-460f-951d-83bb422f39da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26c611be-bb28-460f-951d-83bb422f39da" (UID: "26c611be-bb28-460f-951d-83bb422f39da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:54:53.135642 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.135620    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2ec305-9152-43a2-b3c3-56ad9e271632-kube-api-access-gb4gd" (OuterVolumeSpecName: "kube-api-access-gb4gd") pod "5d2ec305-9152-43a2-b3c3-56ad9e271632" (UID: "5d2ec305-9152-43a2-b3c3-56ad9e271632"). InnerVolumeSpecName "kube-api-access-gb4gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:54:53.135709 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.135644    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c611be-bb28-460f-951d-83bb422f39da-kube-api-access-rhzw4" (OuterVolumeSpecName: "kube-api-access-rhzw4") pod "26c611be-bb28-460f-951d-83bb422f39da" (UID: "26c611be-bb28-460f-951d-83bb422f39da"). InnerVolumeSpecName "kube-api-access-rhzw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:54:53.236514 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.236407    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c611be-bb28-460f-951d-83bb422f39da-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:54:53.236514 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.236437    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhzw4\" (UniqueName: \"kubernetes.io/projected/26c611be-bb28-460f-951d-83bb422f39da-kube-api-access-rhzw4\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:54:53.236514 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.236496    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d2ec305-9152-43a2-b3c3-56ad9e271632-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:54:53.236725 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.236519    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4gd\" (UniqueName: \"kubernetes.io/projected/5d2ec305-9152-43a2-b3c3-56ad9e271632-kube-api-access-gb4gd\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:54:53.549557 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.549445    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:53.947691 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.947618    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"
Sep 08 11:54:53.947691 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.947639    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh" event={"ID":"26c611be-bb28-460f-951d-83bb422f39da","Type":"ContainerDied","Data":"5a873d140826af4e6711dc9a621aa8e39541ce47a8ba25c3061890806e5b72e2"}
Sep 08 11:54:53.948171 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.947727    2561 scope.go:117] "RemoveContainer" containerID="60edb3d19c94fac21a6d2543a2a3bea60510f9f6fc3ddef46224b345af57f06b"
Sep 08 11:54:53.949125 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.949108    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"
Sep 08 11:54:53.949226 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.949120    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2" event={"ID":"5d2ec305-9152-43a2-b3c3-56ad9e271632","Type":"ContainerDied","Data":"af64befc3e71e0a3edd637c59d39b0794327869c6c44ae2a081c79e864dace5f"}
Sep 08 11:54:53.963450 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.963407    2561 scope.go:117] "RemoveContainer" containerID="57611255b96323ca08f47e41236a7b20516afde0738cb61f815b9436219dcc17"
Sep 08 11:54:53.965129 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:54:53.965108    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60edb3d19c94fac21a6d2543a2a3bea60510f9f6fc3ddef46224b345af57f06b\": container with ID starting with 60edb3d19c94fac21a6d2543a2a3bea60510f9f6fc3ddef46224b345af57f06b not found: ID does not exist" containerID="60edb3d19c94fac21a6d2543a2a3bea60510f9f6fc3ddef46224b345af57f06b"
Sep 08 11:54:53.965211 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.965141    2561 image_gc_manager.go:222] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"60edb3d19c94fac21a6d2543a2a3bea60510f9f6fc3ddef46224b345af57f06b\": container with ID starting with 60edb3d19c94fac21a6d2543a2a3bea60510f9f6fc3ddef46224b345af57f06b not found: ID does not exist"
Sep 08 11:54:53.965211 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.965140    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3451748c-06cc-426b-81a7-a33cfe7b4dc9" path="/var/lib/kubelet/pods/3451748c-06cc-426b-81a7-a33cfe7b4dc9/volumes"
Sep 08 11:54:53.977254 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.977232    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"]
Sep 08 11:54:53.982653 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.982632    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77fac0-5ddcb9677d-5vhkh"]
Sep 08 11:54:53.994984 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.994959    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"]
Sep 08 11:54:53.997038 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:53.997020    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77fac0-6d6f65f447-xxcb2"]
Sep 08 11:54:54.230826 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:54.230797    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r_56116325-83f2-4cbf-86d7-3348157096a0/payload-processing/0.log"
Sep 08 11:54:55.960273 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:55.960237    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26c611be-bb28-460f-951d-83bb422f39da" path="/var/lib/kubelet/pods/26c611be-bb28-460f-951d-83bb422f39da/volumes"
Sep 08 11:54:55.960733 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:55.960713    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d2ec305-9152-43a2-b3c3-56ad9e271632" path="/var/lib/kubelet/pods/5d2ec305-9152-43a2-b3c3-56ad9e271632/volumes"
Sep 08 11:54:58.226224 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.226194    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"]
Sep 08 11:54:58.226593 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.226570    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2ec305-9152-43a2-b3c3-56ad9e271632" containerName="payload-processing"
Sep 08 11:54:58.226593 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.226583    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2ec305-9152-43a2-b3c3-56ad9e271632" containerName="payload-processing"
Sep 08 11:54:58.226593 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.226589    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c611be-bb28-460f-951d-83bb422f39da" containerName="payload-pre-processing"
Sep 08 11:54:58.226593 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.226595    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c611be-bb28-460f-951d-83bb422f39da" containerName="payload-pre-processing"
Sep 08 11:54:58.226734 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.226614    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3451748c-06cc-426b-81a7-a33cfe7b4dc9" containerName="revoke-keys"
Sep 08 11:54:58.226734 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.226620    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3451748c-06cc-426b-81a7-a33cfe7b4dc9" containerName="revoke-keys"
Sep 08 11:54:58.226734 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.226675    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3451748c-06cc-426b-81a7-a33cfe7b4dc9" containerName="revoke-keys"
Sep 08 11:54:58.226734 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.226687    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2ec305-9152-43a2-b3c3-56ad9e271632" containerName="payload-processing"
Sep 08 11:54:58.226734 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.226695    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c611be-bb28-460f-951d-83bb422f39da" containerName="payload-pre-processing"
Sep 08 11:54:58.230013 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.229996    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.234209 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.234189    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3f9288\"/\"default-dockercfg-fvc45\""
Sep 08 11:54:58.234329 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.234190    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3f9288\"/\"ipp-route-3f9288-kserve-self-signed-certs\""
Sep 08 11:54:58.234554 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.234539    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3f9288\"/\"kube-root-ca.crt\""
Sep 08 11:54:58.234655 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.234638    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3f9288\"/\"openshift-service-ca.crt\""
Sep 08 11:54:58.241349 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.241328    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"]
Sep 08 11:54:58.251697 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.251666    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:54:58.279647 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.279616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-tls-certs\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.279806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.279665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-home\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.279806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.279684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-dshm\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.279806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.279764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-tmp-dir\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.279806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.279799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-model-cache\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.279955 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.279836    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mctgw\" (UniqueName: \"kubernetes.io/projected/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-kube-api-access-mctgw\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.380373 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.380338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-tmp-dir\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.380373 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.380381    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-model-cache\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.380644 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.380414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mctgw\" (UniqueName: \"kubernetes.io/projected/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-kube-api-access-mctgw\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.380644 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.380457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-tls-certs\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.380644 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.380530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-home\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.380644 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.380555    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-dshm\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.380859 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.380826    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-tmp-dir\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.380914 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.380862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-model-cache\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.380914 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.380890    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-home\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.382761 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.382738    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-dshm\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.382927 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.382909    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-tls-certs\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.389103 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.389085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mctgw\" (UniqueName: \"kubernetes.io/projected/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-kube-api-access-mctgw\") pod \"ipp-route-3f9288-kserve-5b567ddb95-z88m6\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") " pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.539270 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.539178    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:54:58.812556 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.812500    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"]
Sep 08 11:54:58.814772 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:54:58.814747    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9aec16f5_ba8c_42bc_b991_e5dc5b3b21ca.slice/crio-b19c58ca91e4a34bff154fd32fb8007552e4d911d26af3179f241e696f88fbf5 WatchSource:0}: Error finding container b19c58ca91e4a34bff154fd32fb8007552e4d911d26af3179f241e696f88fbf5: Status 404 returned error can't find the container with id b19c58ca91e4a34bff154fd32fb8007552e4d911d26af3179f241e696f88fbf5
Sep 08 11:54:58.974662 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.974627    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6" event={"ID":"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca","Type":"ContainerStarted","Data":"4c47b21f8a4aa941bc22054caf0e89986f5ae4ee456698b0540e5f8cda425e43"}
Sep 08 11:54:58.974827 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.974672    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6" event={"ID":"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca","Type":"ContainerStarted","Data":"b19c58ca91e4a34bff154fd32fb8007552e4d911d26af3179f241e696f88fbf5"}
Sep 08 11:54:58.994673 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:54:58.994623    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6" podStartSLOduration=0.994607737 podStartE2EDuration="994.607737ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:54:58.993319681 +0000 UTC m=+1505.560080239" watchObservedRunningTime="2026-09-08 11:54:58.994607737 +0000 UTC m=+1505.561368278"
Sep 08 11:55:00.298829 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:00.298794    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:00.363278 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:00.363248    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:00.578224 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:00.578120    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:00.913642 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:00.913549    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9"]
Sep 08 11:55:00.917859 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:00.917835    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9"
Sep 08 11:55:00.920759 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:00.920737    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:55:00.920874 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:00.920743    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:55:00.924759 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:00.924735    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9"]
Sep 08 11:55:01.007622 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:01.007580    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa789b1-76c5-48fc-877a-18dce77793a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca8280-vxng9\" (UID: \"8fa789b1-76c5-48fc-877a-18dce77793a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9"
Sep 08 11:55:01.040237 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:01.040201    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:01.108701 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:01.108662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa789b1-76c5-48fc-877a-18dce77793a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca8280-vxng9\" (UID: \"8fa789b1-76c5-48fc-877a-18dce77793a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9"
Sep 08 11:55:01.109336 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:01.109310    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa789b1-76c5-48fc-877a-18dce77793a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca8280-vxng9\" (UID: \"8fa789b1-76c5-48fc-877a-18dce77793a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9"
Sep 08 11:55:01.231493 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:01.231432    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9"
Sep 08 11:55:01.506426 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:01.506371    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9"]
Sep 08 11:55:01.508580 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:55:01.508551    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa789b1_76c5_48fc_877a_18dce77793a8.slice/crio-7d126f6130d9253ddc79ff1ab66c555148844599fd82680ca5a9e591a6b61a76 WatchSource:0}: Error finding container 7d126f6130d9253ddc79ff1ab66c555148844599fd82680ca5a9e591a6b61a76: Status 404 returned error can't find the container with id 7d126f6130d9253ddc79ff1ab66c555148844599fd82680ca5a9e591a6b61a76
Sep 08 11:55:01.990657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:01.990613    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9" event={"ID":"8fa789b1-76c5-48fc-877a-18dce77793a8","Type":"ContainerStarted","Data":"afe2134ee190c357cc910d51078c813260fc5c0d2eced9c91ca64cd8caab5fb1"}
Sep 08 11:55:01.990859 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:01.990671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9" event={"ID":"8fa789b1-76c5-48fc-877a-18dce77793a8","Type":"ContainerStarted","Data":"7d126f6130d9253ddc79ff1ab66c555148844599fd82680ca5a9e591a6b61a76"}
Sep 08 11:55:02.013287 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:02.013234    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9" podStartSLOduration=2.013217921 podStartE2EDuration="2.013217921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:55:02.010503726 +0000 UTC m=+1508.577264263" watchObservedRunningTime="2026-09-08 11:55:02.013217921 +0000 UTC m=+1508.579978462"
Sep 08 11:55:02.996753 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:02.996722    2561 generic.go:356] "Generic (PLEG): container finished" podID="8fa789b1-76c5-48fc-877a-18dce77793a8" containerID="afe2134ee190c357cc910d51078c813260fc5c0d2eced9c91ca64cd8caab5fb1" exitCode=0
Sep 08 11:55:02.997144 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:02.996812    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9" event={"ID":"8fa789b1-76c5-48fc-877a-18dce77793a8","Type":"ContainerDied","Data":"afe2134ee190c357cc910d51078c813260fc5c0d2eced9c91ca64cd8caab5fb1"}
Sep 08 11:55:04.136803 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:04.136775    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9"
Sep 08 11:55:04.234216 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:04.234177    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa789b1-76c5-48fc-877a-18dce77793a8-maas-api-service-ca\") pod \"8fa789b1-76c5-48fc-877a-18dce77793a8\" (UID: \"8fa789b1-76c5-48fc-877a-18dce77793a8\") "
Sep 08 11:55:04.234592 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:04.234567    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa789b1-76c5-48fc-877a-18dce77793a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8fa789b1-76c5-48fc-877a-18dce77793a8" (UID: "8fa789b1-76c5-48fc-877a-18dce77793a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:55:04.334985 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:04.334890    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa789b1-76c5-48fc-877a-18dce77793a8-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:55:05.007240 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:05.007204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9" event={"ID":"8fa789b1-76c5-48fc-877a-18dce77793a8","Type":"ContainerDied","Data":"7d126f6130d9253ddc79ff1ab66c555148844599fd82680ca5a9e591a6b61a76"}
Sep 08 11:55:05.007384 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:05.007262    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d126f6130d9253ddc79ff1ab66c555148844599fd82680ca5a9e591a6b61a76"
Sep 08 11:55:05.007384 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:05.007213    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9"
Sep 08 11:55:08.539793 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:08.539749    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:55:08.540277 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:08.539810    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:55:08.552343 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:08.552315    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:55:09.037363 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:09.037335    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:55:16.461540 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.461445    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:16.505395 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.505342    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"]
Sep 08 11:55:16.505667 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.505644    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc" podUID="56749441-a582-403b-95c8-fb680a7eb178" containerName="payload-processing" containerID="cri-o://39985ab27473220b355def23de94a9e583c08705c00a8433647059dd0969f2ab" gracePeriod=30
Sep 08 11:55:16.510959 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.510931    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"]
Sep 08 11:55:16.511190 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.511163    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99" podUID="0c327595-0cae-4ee3-adaa-9b3db6059cbb" containerName="payload-pre-processing" containerID="cri-o://daf7fe79137c71b0c7267d34c921645ac19a55eb6ed5acd57f3f7f630bbd4924" gracePeriod=30
Sep 08 11:55:16.684504 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.684357    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9"]
Sep 08 11:55:16.686528 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.686483    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca8280-vxng9"]
Sep 08 11:55:16.692828 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.692796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:16.752314 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.752288    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:16.759545 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.759519    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"
Sep 08 11:55:16.791571 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.791549    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"
Sep 08 11:55:16.847570 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.847522    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c327595-0cae-4ee3-adaa-9b3db6059cbb-plugins-config-volume\") pod \"0c327595-0cae-4ee3-adaa-9b3db6059cbb\" (UID: \"0c327595-0cae-4ee3-adaa-9b3db6059cbb\") "
Sep 08 11:55:16.847570 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.847572    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb5mr\" (UniqueName: \"kubernetes.io/projected/56749441-a582-403b-95c8-fb680a7eb178-kube-api-access-cb5mr\") pod \"56749441-a582-403b-95c8-fb680a7eb178\" (UID: \"56749441-a582-403b-95c8-fb680a7eb178\") "
Sep 08 11:55:16.847826 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.847599    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56749441-a582-403b-95c8-fb680a7eb178-plugins-config-volume\") pod \"56749441-a582-403b-95c8-fb680a7eb178\" (UID: \"56749441-a582-403b-95c8-fb680a7eb178\") "
Sep 08 11:55:16.847826 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.847616    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5dfc\" (UniqueName: \"kubernetes.io/projected/0c327595-0cae-4ee3-adaa-9b3db6059cbb-kube-api-access-t5dfc\") pod \"0c327595-0cae-4ee3-adaa-9b3db6059cbb\" (UID: \"0c327595-0cae-4ee3-adaa-9b3db6059cbb\") "
Sep 08 11:55:16.848001 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.847976    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56749441-a582-403b-95c8-fb680a7eb178-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56749441-a582-403b-95c8-fb680a7eb178" (UID: "56749441-a582-403b-95c8-fb680a7eb178"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:55:16.848056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.847985    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c327595-0cae-4ee3-adaa-9b3db6059cbb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c327595-0cae-4ee3-adaa-9b3db6059cbb" (UID: "0c327595-0cae-4ee3-adaa-9b3db6059cbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:55:16.849761 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.849737    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c327595-0cae-4ee3-adaa-9b3db6059cbb-kube-api-access-t5dfc" (OuterVolumeSpecName: "kube-api-access-t5dfc") pod "0c327595-0cae-4ee3-adaa-9b3db6059cbb" (UID: "0c327595-0cae-4ee3-adaa-9b3db6059cbb"). InnerVolumeSpecName "kube-api-access-t5dfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:55:16.849854 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.849768    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56749441-a582-403b-95c8-fb680a7eb178-kube-api-access-cb5mr" (OuterVolumeSpecName: "kube-api-access-cb5mr") pod "56749441-a582-403b-95c8-fb680a7eb178" (UID: "56749441-a582-403b-95c8-fb680a7eb178"). InnerVolumeSpecName "kube-api-access-cb5mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:55:16.949092 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.949061    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c327595-0cae-4ee3-adaa-9b3db6059cbb-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:55:16.949092 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.949089    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb5mr\" (UniqueName: \"kubernetes.io/projected/56749441-a582-403b-95c8-fb680a7eb178-kube-api-access-cb5mr\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:55:16.949092 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.949099    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56749441-a582-403b-95c8-fb680a7eb178-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:55:16.949324 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:16.949108    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5dfc\" (UniqueName: \"kubernetes.io/projected/0c327595-0cae-4ee3-adaa-9b3db6059cbb-kube-api-access-t5dfc\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:55:17.072879 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.072775    2561 generic.go:356] "Generic (PLEG): container finished" podID="56749441-a582-403b-95c8-fb680a7eb178" containerID="39985ab27473220b355def23de94a9e583c08705c00a8433647059dd0969f2ab" exitCode=0
Sep 08 11:55:17.072879 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.072820    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"
Sep 08 11:55:17.072879 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.072854    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc" event={"ID":"56749441-a582-403b-95c8-fb680a7eb178","Type":"ContainerDied","Data":"39985ab27473220b355def23de94a9e583c08705c00a8433647059dd0969f2ab"}
Sep 08 11:55:17.073140 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.072901    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc" event={"ID":"56749441-a582-403b-95c8-fb680a7eb178","Type":"ContainerDied","Data":"5938af7dcad1bc61b64b5cbf34c827c9d9c3428d8eab7c5727de89d241d3054d"}
Sep 08 11:55:17.073140 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.072925    2561 scope.go:117] "RemoveContainer" containerID="39985ab27473220b355def23de94a9e583c08705c00a8433647059dd0969f2ab"
Sep 08 11:55:17.074204 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.074181    2561 generic.go:356] "Generic (PLEG): container finished" podID="0c327595-0cae-4ee3-adaa-9b3db6059cbb" containerID="daf7fe79137c71b0c7267d34c921645ac19a55eb6ed5acd57f3f7f630bbd4924" exitCode=0
Sep 08 11:55:17.074286 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.074212    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"
Sep 08 11:55:17.074286 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.074264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99" event={"ID":"0c327595-0cae-4ee3-adaa-9b3db6059cbb","Type":"ContainerDied","Data":"daf7fe79137c71b0c7267d34c921645ac19a55eb6ed5acd57f3f7f630bbd4924"}
Sep 08 11:55:17.074393 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.074322    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99" event={"ID":"0c327595-0cae-4ee3-adaa-9b3db6059cbb","Type":"ContainerDied","Data":"a354af7a27b857d64e4b7034d6beb3046d67682011190cd5cb673f97b8843ea0"}
Sep 08 11:55:17.083140 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.083121    2561 scope.go:117] "RemoveContainer" containerID="39985ab27473220b355def23de94a9e583c08705c00a8433647059dd0969f2ab"
Sep 08 11:55:17.083434 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:55:17.083415    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39985ab27473220b355def23de94a9e583c08705c00a8433647059dd0969f2ab\": container with ID starting with 39985ab27473220b355def23de94a9e583c08705c00a8433647059dd0969f2ab not found: ID does not exist" containerID="39985ab27473220b355def23de94a9e583c08705c00a8433647059dd0969f2ab"
Sep 08 11:55:17.083559 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.083441    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39985ab27473220b355def23de94a9e583c08705c00a8433647059dd0969f2ab"} err="failed to get container status \"39985ab27473220b355def23de94a9e583c08705c00a8433647059dd0969f2ab\": rpc error: code = NotFound desc = could not find container \"39985ab27473220b355def23de94a9e583c08705c00a8433647059dd0969f2ab\": container with ID starting with 39985ab27473220b355def23de94a9e583c08705c00a8433647059dd0969f2ab not found: ID does not exist"
Sep 08 11:55:17.083559 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.083456    2561 scope.go:117] "RemoveContainer" containerID="daf7fe79137c71b0c7267d34c921645ac19a55eb6ed5acd57f3f7f630bbd4924"
Sep 08 11:55:17.091657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.091634    2561 scope.go:117] "RemoveContainer" containerID="daf7fe79137c71b0c7267d34c921645ac19a55eb6ed5acd57f3f7f630bbd4924"
Sep 08 11:55:17.091922 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:55:17.091900    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf7fe79137c71b0c7267d34c921645ac19a55eb6ed5acd57f3f7f630bbd4924\": container with ID starting with daf7fe79137c71b0c7267d34c921645ac19a55eb6ed5acd57f3f7f630bbd4924 not found: ID does not exist" containerID="daf7fe79137c71b0c7267d34c921645ac19a55eb6ed5acd57f3f7f630bbd4924"
Sep 08 11:55:17.092013 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.091931    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf7fe79137c71b0c7267d34c921645ac19a55eb6ed5acd57f3f7f630bbd4924"} err="failed to get container status \"daf7fe79137c71b0c7267d34c921645ac19a55eb6ed5acd57f3f7f630bbd4924\": rpc error: code = NotFound desc = could not find container \"daf7fe79137c71b0c7267d34c921645ac19a55eb6ed5acd57f3f7f630bbd4924\": container with ID starting with daf7fe79137c71b0c7267d34c921645ac19a55eb6ed5acd57f3f7f630bbd4924 not found: ID does not exist"
Sep 08 11:55:17.100814 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.100791    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"]
Sep 08 11:55:17.103732 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.103711    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca8280-7f54f7c957-rg8zc"]
Sep 08 11:55:17.115555 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.115533    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"]
Sep 08 11:55:17.117974 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.117951    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca8280-6c7db9f7c8-f6s99"]
Sep 08 11:55:17.487404 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.487376    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:17.961124 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.961086    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c327595-0cae-4ee3-adaa-9b3db6059cbb" path="/var/lib/kubelet/pods/0c327595-0cae-4ee3-adaa-9b3db6059cbb/volumes"
Sep 08 11:55:17.961537 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.961516    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56749441-a582-403b-95c8-fb680a7eb178" path="/var/lib/kubelet/pods/56749441-a582-403b-95c8-fb680a7eb178/volumes"
Sep 08 11:55:17.961877 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:17.961860    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa789b1-76c5-48fc-877a-18dce77793a8" path="/var/lib/kubelet/pods/8fa789b1-76c5-48fc-877a-18dce77793a8/volumes"
Sep 08 11:55:23.572037 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:23.571996    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:23.675243 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:23.675205    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:23.862954 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:23.862876    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:24.187968 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.187879    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754"]
Sep 08 11:55:24.188247 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.188232    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa789b1-76c5-48fc-877a-18dce77793a8" containerName="revoke-keys"
Sep 08 11:55:24.188298 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.188248    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa789b1-76c5-48fc-877a-18dce77793a8" containerName="revoke-keys"
Sep 08 11:55:24.188298 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.188258    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56749441-a582-403b-95c8-fb680a7eb178" containerName="payload-processing"
Sep 08 11:55:24.188298 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.188264    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="56749441-a582-403b-95c8-fb680a7eb178" containerName="payload-processing"
Sep 08 11:55:24.188298 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.188283    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c327595-0cae-4ee3-adaa-9b3db6059cbb" containerName="payload-pre-processing"
Sep 08 11:55:24.188298 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.188288    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c327595-0cae-4ee3-adaa-9b3db6059cbb" containerName="payload-pre-processing"
Sep 08 11:55:24.188460 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.188347    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="56749441-a582-403b-95c8-fb680a7eb178" containerName="payload-processing"
Sep 08 11:55:24.188460 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.188356    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa789b1-76c5-48fc-877a-18dce77793a8" containerName="revoke-keys"
Sep 08 11:55:24.188460 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.188361    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c327595-0cae-4ee3-adaa-9b3db6059cbb" containerName="payload-pre-processing"
Sep 08 11:55:24.192451 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.192433    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754"
Sep 08 11:55:24.195246 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.195224    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:55:24.195371 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.195268    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:55:24.199860 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.199833    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754"]
Sep 08 11:55:24.308528 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.308485    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5506d845-528a-46fc-8de5-63b83fafa4af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754\" (UID: \"5506d845-528a-46fc-8de5-63b83fafa4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754"
Sep 08 11:55:24.347568 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.347537    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:24.409886 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.409860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5506d845-528a-46fc-8de5-63b83fafa4af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754\" (UID: \"5506d845-528a-46fc-8de5-63b83fafa4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754"
Sep 08 11:55:24.410502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.410452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5506d845-528a-46fc-8de5-63b83fafa4af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754\" (UID: \"5506d845-528a-46fc-8de5-63b83fafa4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754"
Sep 08 11:55:24.506551 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.506523    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754"
Sep 08 11:55:24.776341 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:24.776317    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754"]
Sep 08 11:55:24.778237 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:55:24.778211    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5506d845_528a_46fc_8de5_63b83fafa4af.slice/crio-122916ae4ec6c16d9dec29d3038a24bc6b8958e87125055cbb7aa273a7c03229 WatchSource:0}: Error finding container 122916ae4ec6c16d9dec29d3038a24bc6b8958e87125055cbb7aa273a7c03229: Status 404 returned error can't find the container with id 122916ae4ec6c16d9dec29d3038a24bc6b8958e87125055cbb7aa273a7c03229
Sep 08 11:55:25.116006 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:25.115924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754" event={"ID":"5506d845-528a-46fc-8de5-63b83fafa4af","Type":"ContainerStarted","Data":"59e4294fcb9c500a1a12a371e4c52ddb6b3e7e711d943b5e30e3019b0c1cfb11"}
Sep 08 11:55:25.116006 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:25.115974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754" event={"ID":"5506d845-528a-46fc-8de5-63b83fafa4af","Type":"ContainerStarted","Data":"122916ae4ec6c16d9dec29d3038a24bc6b8958e87125055cbb7aa273a7c03229"}
Sep 08 11:55:25.143408 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:25.143343    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754" podStartSLOduration=1.143327597 podStartE2EDuration="1.143327597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:55:25.142302433 +0000 UTC m=+1531.709062987" watchObservedRunningTime="2026-09-08 11:55:25.143327597 +0000 UTC m=+1531.710088138"
Sep 08 11:55:26.127405 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:26.127372    2561 generic.go:356] "Generic (PLEG): container finished" podID="5506d845-528a-46fc-8de5-63b83fafa4af" containerID="59e4294fcb9c500a1a12a371e4c52ddb6b3e7e711d943b5e30e3019b0c1cfb11" exitCode=0
Sep 08 11:55:26.127856 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:26.127431    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754" event={"ID":"5506d845-528a-46fc-8de5-63b83fafa4af","Type":"ContainerDied","Data":"59e4294fcb9c500a1a12a371e4c52ddb6b3e7e711d943b5e30e3019b0c1cfb11"}
Sep 08 11:55:27.261327 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:27.261301    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754"
Sep 08 11:55:27.335056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:27.335025    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5506d845-528a-46fc-8de5-63b83fafa4af-maas-api-service-ca\") pod \"5506d845-528a-46fc-8de5-63b83fafa4af\" (UID: \"5506d845-528a-46fc-8de5-63b83fafa4af\") "
Sep 08 11:55:27.335372 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:27.335348    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5506d845-528a-46fc-8de5-63b83fafa4af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5506d845-528a-46fc-8de5-63b83fafa4af" (UID: "5506d845-528a-46fc-8de5-63b83fafa4af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:55:27.436149 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:27.436067    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5506d845-528a-46fc-8de5-63b83fafa4af-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:55:28.140366 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:28.140253    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754"
Sep 08 11:55:28.140366 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:28.140284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754" event={"ID":"5506d845-528a-46fc-8de5-63b83fafa4af","Type":"ContainerDied","Data":"122916ae4ec6c16d9dec29d3038a24bc6b8958e87125055cbb7aa273a7c03229"}
Sep 08 11:55:28.140366 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:28.140336    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="122916ae4ec6c16d9dec29d3038a24bc6b8958e87125055cbb7aa273a7c03229"
Sep 08 11:55:34.742148 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:34.742117    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"]
Sep 08 11:55:34.742624 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:34.742380    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk" podUID="0cd702a7-54b6-47be-93b9-9b2d13792e4e" containerName="payload-processing" containerID="cri-o://1b62542f1a5d309dafab39d17660de85b49f1ae4b88edba15e2d593d57f7fdfb" gracePeriod=30
Sep 08 11:55:34.750526 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:34.750499    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"]
Sep 08 11:55:34.750736 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:34.750713    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g" podUID="11d11076-e0eb-42eb-b785-714ca7611eda" containerName="payload-pre-processing" containerID="cri-o://1f311ee8dfc5911c2bfd747b914bf1c27a16b97499665ea0007ad6fed6145cef" gracePeriod=30
Sep 08 11:55:34.825927 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:34.825900    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:34.941876 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:34.941620    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754"]
Sep 08 11:55:34.942697 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:34.942613    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:34.943029 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:34.943009    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-69606d-l6754"]
Sep 08 11:55:35.028822 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.028793    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:35.029146 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.029124    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"
Sep 08 11:55:35.031931 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.031915    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"
Sep 08 11:55:35.174381 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.174350    2561 generic.go:356] "Generic (PLEG): container finished" podID="0cd702a7-54b6-47be-93b9-9b2d13792e4e" containerID="1b62542f1a5d309dafab39d17660de85b49f1ae4b88edba15e2d593d57f7fdfb" exitCode=0
Sep 08 11:55:35.174585 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.174403    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"
Sep 08 11:55:35.174585 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.174451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk" event={"ID":"0cd702a7-54b6-47be-93b9-9b2d13792e4e","Type":"ContainerDied","Data":"1b62542f1a5d309dafab39d17660de85b49f1ae4b88edba15e2d593d57f7fdfb"}
Sep 08 11:55:35.174585 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.174524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk" event={"ID":"0cd702a7-54b6-47be-93b9-9b2d13792e4e","Type":"ContainerDied","Data":"1a00d98837bcd96cd3b026deed78fe6feb184fe5c65860be09f9a9aa1e4593e6"}
Sep 08 11:55:35.174585 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.174558    2561 scope.go:117] "RemoveContainer" containerID="1b62542f1a5d309dafab39d17660de85b49f1ae4b88edba15e2d593d57f7fdfb"
Sep 08 11:55:35.175867 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.175846    2561 generic.go:356] "Generic (PLEG): container finished" podID="11d11076-e0eb-42eb-b785-714ca7611eda" containerID="1f311ee8dfc5911c2bfd747b914bf1c27a16b97499665ea0007ad6fed6145cef" exitCode=0
Sep 08 11:55:35.175966 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.175873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g" event={"ID":"11d11076-e0eb-42eb-b785-714ca7611eda","Type":"ContainerDied","Data":"1f311ee8dfc5911c2bfd747b914bf1c27a16b97499665ea0007ad6fed6145cef"}
Sep 08 11:55:35.175966 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.175932    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g" event={"ID":"11d11076-e0eb-42eb-b785-714ca7611eda","Type":"ContainerDied","Data":"c77c74632ab0366b7447c511e2512ae7d7176736177f2bbda80c145c1dab72fe"}
Sep 08 11:55:35.175966 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.175884    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"
Sep 08 11:55:35.184457 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.184433    2561 scope.go:117] "RemoveContainer" containerID="1b62542f1a5d309dafab39d17660de85b49f1ae4b88edba15e2d593d57f7fdfb"
Sep 08 11:55:35.184765 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:55:35.184747    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b62542f1a5d309dafab39d17660de85b49f1ae4b88edba15e2d593d57f7fdfb\": container with ID starting with 1b62542f1a5d309dafab39d17660de85b49f1ae4b88edba15e2d593d57f7fdfb not found: ID does not exist" containerID="1b62542f1a5d309dafab39d17660de85b49f1ae4b88edba15e2d593d57f7fdfb"
Sep 08 11:55:35.184825 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.184773    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b62542f1a5d309dafab39d17660de85b49f1ae4b88edba15e2d593d57f7fdfb"} err="failed to get container status \"1b62542f1a5d309dafab39d17660de85b49f1ae4b88edba15e2d593d57f7fdfb\": rpc error: code = NotFound desc = could not find container \"1b62542f1a5d309dafab39d17660de85b49f1ae4b88edba15e2d593d57f7fdfb\": container with ID starting with 1b62542f1a5d309dafab39d17660de85b49f1ae4b88edba15e2d593d57f7fdfb not found: ID does not exist"
Sep 08 11:55:35.184825 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.184791    2561 scope.go:117] "RemoveContainer" containerID="1f311ee8dfc5911c2bfd747b914bf1c27a16b97499665ea0007ad6fed6145cef"
Sep 08 11:55:35.193514 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.193495    2561 scope.go:117] "RemoveContainer" containerID="1f311ee8dfc5911c2bfd747b914bf1c27a16b97499665ea0007ad6fed6145cef"
Sep 08 11:55:35.193815 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:55:35.193794    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f311ee8dfc5911c2bfd747b914bf1c27a16b97499665ea0007ad6fed6145cef\": container with ID starting with 1f311ee8dfc5911c2bfd747b914bf1c27a16b97499665ea0007ad6fed6145cef not found: ID does not exist" containerID="1f311ee8dfc5911c2bfd747b914bf1c27a16b97499665ea0007ad6fed6145cef"
Sep 08 11:55:35.193883 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.193822    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f311ee8dfc5911c2bfd747b914bf1c27a16b97499665ea0007ad6fed6145cef"} err="failed to get container status \"1f311ee8dfc5911c2bfd747b914bf1c27a16b97499665ea0007ad6fed6145cef\": rpc error: code = NotFound desc = could not find container \"1f311ee8dfc5911c2bfd747b914bf1c27a16b97499665ea0007ad6fed6145cef\": container with ID starting with 1f311ee8dfc5911c2bfd747b914bf1c27a16b97499665ea0007ad6fed6145cef not found: ID does not exist"
Sep 08 11:55:35.205154 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.205128    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82d8j\" (UniqueName: \"kubernetes.io/projected/11d11076-e0eb-42eb-b785-714ca7611eda-kube-api-access-82d8j\") pod \"11d11076-e0eb-42eb-b785-714ca7611eda\" (UID: \"11d11076-e0eb-42eb-b785-714ca7611eda\") "
Sep 08 11:55:35.205256 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.205207    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd702a7-54b6-47be-93b9-9b2d13792e4e-plugins-config-volume\") pod \"0cd702a7-54b6-47be-93b9-9b2d13792e4e\" (UID: \"0cd702a7-54b6-47be-93b9-9b2d13792e4e\") "
Sep 08 11:55:35.205319 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.205262    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcqqc\" (UniqueName: \"kubernetes.io/projected/0cd702a7-54b6-47be-93b9-9b2d13792e4e-kube-api-access-wcqqc\") pod \"0cd702a7-54b6-47be-93b9-9b2d13792e4e\" (UID: \"0cd702a7-54b6-47be-93b9-9b2d13792e4e\") "
Sep 08 11:55:35.205319 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.205289    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d11076-e0eb-42eb-b785-714ca7611eda-plugins-config-volume\") pod \"11d11076-e0eb-42eb-b785-714ca7611eda\" (UID: \"11d11076-e0eb-42eb-b785-714ca7611eda\") "
Sep 08 11:55:35.205583 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.205552    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd702a7-54b6-47be-93b9-9b2d13792e4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0cd702a7-54b6-47be-93b9-9b2d13792e4e" (UID: "0cd702a7-54b6-47be-93b9-9b2d13792e4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:55:35.205790 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.205766    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d11076-e0eb-42eb-b785-714ca7611eda-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11d11076-e0eb-42eb-b785-714ca7611eda" (UID: "11d11076-e0eb-42eb-b785-714ca7611eda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:55:35.207222 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.207202    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d11076-e0eb-42eb-b785-714ca7611eda-kube-api-access-82d8j" (OuterVolumeSpecName: "kube-api-access-82d8j") pod "11d11076-e0eb-42eb-b785-714ca7611eda" (UID: "11d11076-e0eb-42eb-b785-714ca7611eda"). InnerVolumeSpecName "kube-api-access-82d8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:55:35.207301 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.207233    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd702a7-54b6-47be-93b9-9b2d13792e4e-kube-api-access-wcqqc" (OuterVolumeSpecName: "kube-api-access-wcqqc") pod "0cd702a7-54b6-47be-93b9-9b2d13792e4e" (UID: "0cd702a7-54b6-47be-93b9-9b2d13792e4e"). InnerVolumeSpecName "kube-api-access-wcqqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:55:35.293437 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.293380    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:35.306572 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.306549    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82d8j\" (UniqueName: \"kubernetes.io/projected/11d11076-e0eb-42eb-b785-714ca7611eda-kube-api-access-82d8j\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:55:35.306669 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.306574    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd702a7-54b6-47be-93b9-9b2d13792e4e-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:55:35.306669 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.306588    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcqqc\" (UniqueName: \"kubernetes.io/projected/0cd702a7-54b6-47be-93b9-9b2d13792e4e-kube-api-access-wcqqc\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:55:35.306669 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.306601    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d11076-e0eb-42eb-b785-714ca7611eda-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:55:35.500455 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.500425    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"]
Sep 08 11:55:35.506901 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.506878    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-69606d-5d6f5ccc5-ndzzk"]
Sep 08 11:55:35.518928 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.518902    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"]
Sep 08 11:55:35.523502 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.523480    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-69606d-69d7b67b4b-spp9g"]
Sep 08 11:55:35.960916 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.960886    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cd702a7-54b6-47be-93b9-9b2d13792e4e" path="/var/lib/kubelet/pods/0cd702a7-54b6-47be-93b9-9b2d13792e4e/volumes"
Sep 08 11:55:35.961272 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.961209    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11d11076-e0eb-42eb-b785-714ca7611eda" path="/var/lib/kubelet/pods/11d11076-e0eb-42eb-b785-714ca7611eda/volumes"
Sep 08 11:55:35.961506 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:35.961492    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5506d845-528a-46fc-8de5-63b83fafa4af" path="/var/lib/kubelet/pods/5506d845-528a-46fc-8de5-63b83fafa4af/volumes"
Sep 08 11:55:41.757980 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:41.757946    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:41.829354 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:41.829325    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:42.031372 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.031291    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:42.362325 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.362245    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g"]
Sep 08 11:55:42.362621 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.362608    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d11076-e0eb-42eb-b785-714ca7611eda" containerName="payload-pre-processing"
Sep 08 11:55:42.362682 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.362623    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d11076-e0eb-42eb-b785-714ca7611eda" containerName="payload-pre-processing"
Sep 08 11:55:42.362682 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.362630    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5506d845-528a-46fc-8de5-63b83fafa4af" containerName="revoke-keys"
Sep 08 11:55:42.362682 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.362636    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5506d845-528a-46fc-8de5-63b83fafa4af" containerName="revoke-keys"
Sep 08 11:55:42.362682 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.362644    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd702a7-54b6-47be-93b9-9b2d13792e4e" containerName="payload-processing"
Sep 08 11:55:42.362682 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.362650    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd702a7-54b6-47be-93b9-9b2d13792e4e" containerName="payload-processing"
Sep 08 11:55:42.362840 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.362713    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd702a7-54b6-47be-93b9-9b2d13792e4e" containerName="payload-processing"
Sep 08 11:55:42.362840 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.362723    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d11076-e0eb-42eb-b785-714ca7611eda" containerName="payload-pre-processing"
Sep 08 11:55:42.362840 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.362731    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5506d845-528a-46fc-8de5-63b83fafa4af" containerName="revoke-keys"
Sep 08 11:55:42.367045 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.367027    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g"
Sep 08 11:55:42.369858 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.369839    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:55:42.369976 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.369843    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:55:42.376735 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.376677    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g"]
Sep 08 11:55:42.461843 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.461811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbce5a08-3d4c-4b70-b859-16bfe130c4f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g\" (UID: \"cbce5a08-3d4c-4b70-b859-16bfe130c4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g"
Sep 08 11:55:42.563089 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.563057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbce5a08-3d4c-4b70-b859-16bfe130c4f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g\" (UID: \"cbce5a08-3d4c-4b70-b859-16bfe130c4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g"
Sep 08 11:55:42.563696 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.563677    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbce5a08-3d4c-4b70-b859-16bfe130c4f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g\" (UID: \"cbce5a08-3d4c-4b70-b859-16bfe130c4f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g"
Sep 08 11:55:42.680165 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.680093    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g"
Sep 08 11:55:42.955827 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:42.955803    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g"]
Sep 08 11:55:42.958010 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:55:42.957986    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbce5a08_3d4c_4b70_b859_16bfe130c4f5.slice/crio-5e4345d0621d3d7ee592a92d7b4b7c1a316994de118edcbb437dec110b352720 WatchSource:0}: Error finding container 5e4345d0621d3d7ee592a92d7b4b7c1a316994de118edcbb437dec110b352720: Status 404 returned error can't find the container with id 5e4345d0621d3d7ee592a92d7b4b7c1a316994de118edcbb437dec110b352720
Sep 08 11:55:43.044176 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:43.044142    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:43.218521 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:43.218427    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g" event={"ID":"cbce5a08-3d4c-4b70-b859-16bfe130c4f5","Type":"ContainerStarted","Data":"5e4345d0621d3d7ee592a92d7b4b7c1a316994de118edcbb437dec110b352720"}
Sep 08 11:55:44.023400 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:44.023369    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:44.224391 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:44.224357    2561 generic.go:356] "Generic (PLEG): container finished" podID="cbce5a08-3d4c-4b70-b859-16bfe130c4f5" containerID="c9b7ee24a526d45fa93b9451e16cfd493b674fa542458f61fcac4ba1f8f77c62" exitCode=0
Sep 08 11:55:44.224578 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:44.224446    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g" event={"ID":"cbce5a08-3d4c-4b70-b859-16bfe130c4f5","Type":"ContainerDied","Data":"c9b7ee24a526d45fa93b9451e16cfd493b674fa542458f61fcac4ba1f8f77c62"}
Sep 08 11:55:45.355576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:45.355550    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g"
Sep 08 11:55:45.491141 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:45.491067    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbce5a08-3d4c-4b70-b859-16bfe130c4f5-maas-api-service-ca\") pod \"cbce5a08-3d4c-4b70-b859-16bfe130c4f5\" (UID: \"cbce5a08-3d4c-4b70-b859-16bfe130c4f5\") "
Sep 08 11:55:45.491418 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:45.491389    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbce5a08-3d4c-4b70-b859-16bfe130c4f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cbce5a08-3d4c-4b70-b859-16bfe130c4f5" (UID: "cbce5a08-3d4c-4b70-b859-16bfe130c4f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:55:45.592407 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:45.592381    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbce5a08-3d4c-4b70-b859-16bfe130c4f5-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:55:46.234805 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:46.234723    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g"
Sep 08 11:55:46.234805 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:46.234762    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g" event={"ID":"cbce5a08-3d4c-4b70-b859-16bfe130c4f5","Type":"ContainerDied","Data":"5e4345d0621d3d7ee592a92d7b4b7c1a316994de118edcbb437dec110b352720"}
Sep 08 11:55:46.234983 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:46.234807    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e4345d0621d3d7ee592a92d7b4b7c1a316994de118edcbb437dec110b352720"
Sep 08 11:55:46.521923 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:46.521850    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r_56116325-83f2-4cbf-86d7-3348157096a0/payload-processing/0.log"
Sep 08 11:55:46.809861 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:46.809786    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:47.147929 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:47.147826    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:47.947732 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:47.947698    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:50.389165 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:50.389128    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:50.506184 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:50.506154    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:51.552446 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:51.552418    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:51.622827 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:51.622796    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:51.724950 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:51.724914    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:52.089066 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.089035    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:52.188740 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.188706    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:52.305588 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.305548    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"]
Sep 08 11:55:52.305896 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.305884    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbce5a08-3d4c-4b70-b859-16bfe130c4f5" containerName="revoke-keys"
Sep 08 11:55:52.305946 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.305897    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbce5a08-3d4c-4b70-b859-16bfe130c4f5" containerName="revoke-keys"
Sep 08 11:55:52.305978 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.305959    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbce5a08-3d4c-4b70-b859-16bfe130c4f5" containerName="revoke-keys"
Sep 08 11:55:52.308934 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.308919    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"
Sep 08 11:55:52.312070 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.312047    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-64178b-dockercfg-lq6c8\""
Sep 08 11:55:52.312070 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.312068    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-64178b\""
Sep 08 11:55:52.320198 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.320177    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"]
Sep 08 11:55:52.345290 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.345234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd94394-046f-4ac1-8cf1-77d7f85d0fe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6\" (UID: \"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"
Sep 08 11:55:52.345290 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.345266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdrgf\" (UniqueName: \"kubernetes.io/projected/1dd94394-046f-4ac1-8cf1-77d7f85d0fe1-kube-api-access-vdrgf\") pod \"payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6\" (UID: \"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"
Sep 08 11:55:52.381607 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.381580    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"]
Sep 08 11:55:52.385186 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.385168    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"
Sep 08 11:55:52.398056 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.398026    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"]
Sep 08 11:55:52.446158 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.446125    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b801532-302e-465f-ab8f-4ba6abd9211d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79\" (UID: \"2b801532-302e-465f-ab8f-4ba6abd9211d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"
Sep 08 11:55:52.446325 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.446168    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd94394-046f-4ac1-8cf1-77d7f85d0fe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6\" (UID: \"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"
Sep 08 11:55:52.446325 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.446190    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdrgf\" (UniqueName: \"kubernetes.io/projected/1dd94394-046f-4ac1-8cf1-77d7f85d0fe1-kube-api-access-vdrgf\") pod \"payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6\" (UID: \"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"
Sep 08 11:55:52.446325 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.446220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbtm6\" (UniqueName: \"kubernetes.io/projected/2b801532-302e-465f-ab8f-4ba6abd9211d-kube-api-access-nbtm6\") pod \"payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79\" (UID: \"2b801532-302e-465f-ab8f-4ba6abd9211d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"
Sep 08 11:55:52.446905 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.446881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd94394-046f-4ac1-8cf1-77d7f85d0fe1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6\" (UID: \"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"
Sep 08 11:55:52.463439 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.463416    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdrgf\" (UniqueName: \"kubernetes.io/projected/1dd94394-046f-4ac1-8cf1-77d7f85d0fe1-kube-api-access-vdrgf\") pod \"payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6\" (UID: \"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"
Sep 08 11:55:52.489089 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.489059    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:52.541115 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.541085    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"]
Sep 08 11:55:52.541307 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.541282    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5" podUID="62de157b-2c32-4c30-a40c-d8ffef499253" containerName="payload-pre-processing" containerID="cri-o://2e2cd787661a927d1a536b3a26680d659ce788541ee68e05a1b13dcdb57fe303" gracePeriod=30
Sep 08 11:55:52.547459 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.547431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbtm6\" (UniqueName: \"kubernetes.io/projected/2b801532-302e-465f-ab8f-4ba6abd9211d-kube-api-access-nbtm6\") pod \"payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79\" (UID: \"2b801532-302e-465f-ab8f-4ba6abd9211d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"
Sep 08 11:55:52.547598 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.547584    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b801532-302e-465f-ab8f-4ba6abd9211d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79\" (UID: \"2b801532-302e-465f-ab8f-4ba6abd9211d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"
Sep 08 11:55:52.548415 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.548389    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b801532-302e-465f-ab8f-4ba6abd9211d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79\" (UID: \"2b801532-302e-465f-ab8f-4ba6abd9211d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"
Sep 08 11:55:52.569049 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.569027    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbtm6\" (UniqueName: \"kubernetes.io/projected/2b801532-302e-465f-ab8f-4ba6abd9211d-kube-api-access-nbtm6\") pod \"payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79\" (UID: \"2b801532-302e-465f-ab8f-4ba6abd9211d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"
Sep 08 11:55:52.574938 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.574917    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:52.619562 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.619507    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"
Sep 08 11:55:52.672868 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.672818    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:52.695539 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.695310    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"
Sep 08 11:55:52.774809 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.774780    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:52.779126 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.779103    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g"]
Sep 08 11:55:52.785782 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.785684    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-489cdd-vvw7g"]
Sep 08 11:55:52.806317 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.806296    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"
Sep 08 11:55:52.844586 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.844559    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:52.849617 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.849593    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62de157b-2c32-4c30-a40c-d8ffef499253-plugins-config-volume\") pod \"62de157b-2c32-4c30-a40c-d8ffef499253\" (UID: \"62de157b-2c32-4c30-a40c-d8ffef499253\") "
Sep 08 11:55:52.849740 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.849675    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prfns\" (UniqueName: \"kubernetes.io/projected/62de157b-2c32-4c30-a40c-d8ffef499253-kube-api-access-prfns\") pod \"62de157b-2c32-4c30-a40c-d8ffef499253\" (UID: \"62de157b-2c32-4c30-a40c-d8ffef499253\") "
Sep 08 11:55:52.849958 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.849937    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62de157b-2c32-4c30-a40c-d8ffef499253-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62de157b-2c32-4c30-a40c-d8ffef499253" (UID: "62de157b-2c32-4c30-a40c-d8ffef499253"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:55:52.851753 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.851730    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62de157b-2c32-4c30-a40c-d8ffef499253-kube-api-access-prfns" (OuterVolumeSpecName: "kube-api-access-prfns") pod "62de157b-2c32-4c30-a40c-d8ffef499253" (UID: "62de157b-2c32-4c30-a40c-d8ffef499253"). InnerVolumeSpecName "kube-api-access-prfns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:55:52.897962 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.897934    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"]
Sep 08 11:55:52.899090 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:55:52.899062    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd94394_046f_4ac1_8cf1_77d7f85d0fe1.slice/crio-8ec10748c69506a183d23333b26d980f410e6d6ee66f3ff85d30d4b977930fb9 WatchSource:0}: Error finding container 8ec10748c69506a183d23333b26d980f410e6d6ee66f3ff85d30d4b977930fb9: Status 404 returned error can't find the container with id 8ec10748c69506a183d23333b26d980f410e6d6ee66f3ff85d30d4b977930fb9
Sep 08 11:55:52.951166 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.951142    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prfns\" (UniqueName: \"kubernetes.io/projected/62de157b-2c32-4c30-a40c-d8ffef499253-kube-api-access-prfns\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:55:52.951166 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.951165    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62de157b-2c32-4c30-a40c-d8ffef499253-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:55:52.985598 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:52.985554    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"]
Sep 08 11:55:52.988898 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:55:52.988874    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b801532_302e_465f_ab8f_4ba6abd9211d.slice/crio-52b0f819500ab99c693a59e62c669a31df9224dae240b7205dc99192b266b6ed WatchSource:0}: Error finding container 52b0f819500ab99c693a59e62c669a31df9224dae240b7205dc99192b266b6ed: Status 404 returned error can't find the container with id 52b0f819500ab99c693a59e62c669a31df9224dae240b7205dc99192b266b6ed
Sep 08 11:55:53.145919 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.145888    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:55:53.274395 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.274360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79" event={"ID":"2b801532-302e-465f-ab8f-4ba6abd9211d","Type":"ContainerStarted","Data":"aaec655b619537243eab7c7b09bda3a74b80e85fd3de5776124cd8d574660943"}
Sep 08 11:55:53.274631 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.274417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79" event={"ID":"2b801532-302e-465f-ab8f-4ba6abd9211d","Type":"ContainerStarted","Data":"52b0f819500ab99c693a59e62c669a31df9224dae240b7205dc99192b266b6ed"}
Sep 08 11:55:53.274631 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.274444    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"
Sep 08 11:55:53.276258 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.276233    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6" event={"ID":"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1","Type":"ContainerStarted","Data":"9dc3cfe681b5e50c02cfb3078533c4f9004dfee08404cfbc6f39ef53c22ceeac"}
Sep 08 11:55:53.276386 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.276281    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6" event={"ID":"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1","Type":"ContainerStarted","Data":"8ec10748c69506a183d23333b26d980f410e6d6ee66f3ff85d30d4b977930fb9"}
Sep 08 11:55:53.276386 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.276314    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"
Sep 08 11:55:53.277563 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.277538    2561 generic.go:356] "Generic (PLEG): container finished" podID="62de157b-2c32-4c30-a40c-d8ffef499253" containerID="2e2cd787661a927d1a536b3a26680d659ce788541ee68e05a1b13dcdb57fe303" exitCode=0
Sep 08 11:55:53.277642 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.277592    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5" event={"ID":"62de157b-2c32-4c30-a40c-d8ffef499253","Type":"ContainerDied","Data":"2e2cd787661a927d1a536b3a26680d659ce788541ee68e05a1b13dcdb57fe303"}
Sep 08 11:55:53.277642 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.277635    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5" event={"ID":"62de157b-2c32-4c30-a40c-d8ffef499253","Type":"ContainerDied","Data":"1c3cfd2ba88aeecf184750e898a6527b815c840fb958a0092ea5d901044fe8d6"}
Sep 08 11:55:53.277706 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.277594    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"
Sep 08 11:55:53.277706 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.277656    2561 scope.go:117] "RemoveContainer" containerID="2e2cd787661a927d1a536b3a26680d659ce788541ee68e05a1b13dcdb57fe303"
Sep 08 11:55:53.288547 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.288390    2561 scope.go:117] "RemoveContainer" containerID="2e2cd787661a927d1a536b3a26680d659ce788541ee68e05a1b13dcdb57fe303"
Sep 08 11:55:53.294382 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:55:53.294343    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2cd787661a927d1a536b3a26680d659ce788541ee68e05a1b13dcdb57fe303\": container with ID starting with 2e2cd787661a927d1a536b3a26680d659ce788541ee68e05a1b13dcdb57fe303 not found: ID does not exist" containerID="2e2cd787661a927d1a536b3a26680d659ce788541ee68e05a1b13dcdb57fe303"
Sep 08 11:55:53.294489 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.294392    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2cd787661a927d1a536b3a26680d659ce788541ee68e05a1b13dcdb57fe303"} err="failed to get container status \"2e2cd787661a927d1a536b3a26680d659ce788541ee68e05a1b13dcdb57fe303\": rpc error: code = NotFound desc = could not find container \"2e2cd787661a927d1a536b3a26680d659ce788541ee68e05a1b13dcdb57fe303\": container with ID starting with 2e2cd787661a927d1a536b3a26680d659ce788541ee68e05a1b13dcdb57fe303 not found: ID does not exist"
Sep 08 11:55:53.298392 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.298349    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79" podStartSLOduration=1.298335947 podStartE2EDuration="1.298335947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:55:53.296749873 +0000 UTC m=+1559.863510414" watchObservedRunningTime="2026-09-08 11:55:53.298335947 +0000 UTC m=+1559.865096487"
Sep 08 11:55:53.312951 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.312925    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"]
Sep 08 11:55:53.317147 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.317128    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-489cdd-77cc68c6b-pnbt5"]
Sep 08 11:55:53.333190 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.333150    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6" podStartSLOduration=1.333137702 podStartE2EDuration="1.333137702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:55:53.332173808 +0000 UTC m=+1559.898934354" watchObservedRunningTime="2026-09-08 11:55:53.333137702 +0000 UTC m=+1559.899898242"
Sep 08 11:55:53.961344 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.961311    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62de157b-2c32-4c30-a40c-d8ffef499253" path="/var/lib/kubelet/pods/62de157b-2c32-4c30-a40c-d8ffef499253/volumes"
Sep 08 11:55:53.961826 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:53.961647    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbce5a08-3d4c-4b70-b859-16bfe130c4f5" path="/var/lib/kubelet/pods/cbce5a08-3d4c-4b70-b859-16bfe130c4f5/volumes"
Sep 08 11:55:54.284533 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:54.284443    2561 scope.go:117] "RemoveContainer" containerID="1b5595a576aafc844a2aaa4f91de972857781a7294aeff09c304d81d151d2a2c"
Sep 08 11:55:54.292929 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:55:54.292911    2561 scope.go:117] "RemoveContainer" containerID="dbeff067bfafc743c1705ec7556df96b9c6f3d74859283e5ed89fb2d6f84d69c"
Sep 08 11:56:00.196939 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:00.196896    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:00.309708 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:00.309676    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:04.285106 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:04.285079    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"
Sep 08 11:56:04.285506 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:04.285153    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"
Sep 08 11:56:15.379979 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:15.379934    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:16.089634 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:16.089599    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29"]
Sep 08 11:56:16.089969 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:16.089956    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62de157b-2c32-4c30-a40c-d8ffef499253" containerName="payload-pre-processing"
Sep 08 11:56:16.090014 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:16.089971    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="62de157b-2c32-4c30-a40c-d8ffef499253" containerName="payload-pre-processing"
Sep 08 11:56:16.090050 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:16.090039    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="62de157b-2c32-4c30-a40c-d8ffef499253" containerName="payload-pre-processing"
Sep 08 11:56:16.093203 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:16.093188    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29"
Sep 08 11:56:16.096156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:16.096130    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:56:16.096291 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:16.096209    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:56:16.102827 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:16.102805    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29"]
Sep 08 11:56:16.142097 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:16.142064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9c7f70-045e-4f3e-a040-c9a079e7fdd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29\" (UID: \"0d9c7f70-045e-4f3e-a040-c9a079e7fdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29"
Sep 08 11:56:16.242955 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:16.242915    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9c7f70-045e-4f3e-a040-c9a079e7fdd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29\" (UID: \"0d9c7f70-045e-4f3e-a040-c9a079e7fdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29"
Sep 08 11:56:16.243549 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:16.243532    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9c7f70-045e-4f3e-a040-c9a079e7fdd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29\" (UID: \"0d9c7f70-045e-4f3e-a040-c9a079e7fdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29"
Sep 08 11:56:16.404627 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:16.404559    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29"
Sep 08 11:56:16.675137 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:16.675102    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29"]
Sep 08 11:56:16.676719 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:56:16.676692    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9c7f70_045e_4f3e_a040_c9a079e7fdd5.slice/crio-3c91c5b9b2d60f6d825515261f5dcf7c369797dd18483d8950262b54ca0b3c53 WatchSource:0}: Error finding container 3c91c5b9b2d60f6d825515261f5dcf7c369797dd18483d8950262b54ca0b3c53: Status 404 returned error can't find the container with id 3c91c5b9b2d60f6d825515261f5dcf7c369797dd18483d8950262b54ca0b3c53
Sep 08 11:56:17.396164 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:17.396132    2561 generic.go:356] "Generic (PLEG): container finished" podID="0d9c7f70-045e-4f3e-a040-c9a079e7fdd5" containerID="178a25d65214a70aaafa47b641b568146a401930f2e4e3dfb5e1a7bdb1f8bed2" exitCode=0
Sep 08 11:56:17.396327 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:17.396185    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29" event={"ID":"0d9c7f70-045e-4f3e-a040-c9a079e7fdd5","Type":"ContainerDied","Data":"178a25d65214a70aaafa47b641b568146a401930f2e4e3dfb5e1a7bdb1f8bed2"}
Sep 08 11:56:17.396327 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:17.396228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29" event={"ID":"0d9c7f70-045e-4f3e-a040-c9a079e7fdd5","Type":"ContainerStarted","Data":"3c91c5b9b2d60f6d825515261f5dcf7c369797dd18483d8950262b54ca0b3c53"}
Sep 08 11:56:18.530174 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:18.530152    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29"
Sep 08 11:56:18.562722 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:18.562695    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9c7f70-045e-4f3e-a040-c9a079e7fdd5-maas-api-service-ca\") pod \"0d9c7f70-045e-4f3e-a040-c9a079e7fdd5\" (UID: \"0d9c7f70-045e-4f3e-a040-c9a079e7fdd5\") "
Sep 08 11:56:18.563022 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:18.562998    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d9c7f70-045e-4f3e-a040-c9a079e7fdd5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0d9c7f70-045e-4f3e-a040-c9a079e7fdd5" (UID: "0d9c7f70-045e-4f3e-a040-c9a079e7fdd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:56:18.663736 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:18.663659    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9c7f70-045e-4f3e-a040-c9a079e7fdd5-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:56:19.406919 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:19.406888    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29"
Sep 08 11:56:19.407094 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:19.406894    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29" event={"ID":"0d9c7f70-045e-4f3e-a040-c9a079e7fdd5","Type":"ContainerDied","Data":"3c91c5b9b2d60f6d825515261f5dcf7c369797dd18483d8950262b54ca0b3c53"}
Sep 08 11:56:19.407094 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:19.407009    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c91c5b9b2d60f6d825515261f5dcf7c369797dd18483d8950262b54ca0b3c53"
Sep 08 11:56:22.224268 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.224232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:22.244336 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.244306    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"]
Sep 08 11:56:22.244902 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.244567    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79" podUID="2b801532-302e-465f-ab8f-4ba6abd9211d" containerName="payload-processing" containerID="cri-o://aaec655b619537243eab7c7b09bda3a74b80e85fd3de5776124cd8d574660943" gracePeriod=30
Sep 08 11:56:22.252009 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.251978    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"]
Sep 08 11:56:22.252392 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.252177    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6" podUID="1dd94394-046f-4ac1-8cf1-77d7f85d0fe1" containerName="payload-pre-processing" containerID="cri-o://9dc3cfe681b5e50c02cfb3078533c4f9004dfee08404cfbc6f39ef53c22ceeac" gracePeriod=30
Sep 08 11:56:22.325452 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.325420    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:22.422969 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.422889    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b801532-302e-465f-ab8f-4ba6abd9211d" containerID="aaec655b619537243eab7c7b09bda3a74b80e85fd3de5776124cd8d574660943" exitCode=0
Sep 08 11:56:22.423106 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.422981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79" event={"ID":"2b801532-302e-465f-ab8f-4ba6abd9211d","Type":"ContainerDied","Data":"aaec655b619537243eab7c7b09bda3a74b80e85fd3de5776124cd8d574660943"}
Sep 08 11:56:22.424382 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.424356    2561 generic.go:356] "Generic (PLEG): container finished" podID="1dd94394-046f-4ac1-8cf1-77d7f85d0fe1" containerID="9dc3cfe681b5e50c02cfb3078533c4f9004dfee08404cfbc6f39ef53c22ceeac" exitCode=0
Sep 08 11:56:22.424499 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.424396    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6" event={"ID":"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1","Type":"ContainerDied","Data":"9dc3cfe681b5e50c02cfb3078533c4f9004dfee08404cfbc6f39ef53c22ceeac"}
Sep 08 11:56:22.428657 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.428633    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:22.430598 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.430577    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29"]
Sep 08 11:56:22.433390 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.433369    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-64178b-m7q29"]
Sep 08 11:56:22.523830 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.523807    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"
Sep 08 11:56:22.525381 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.525360    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:22.526524 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.526511    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"
Sep 08 11:56:22.599401 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.599375    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdrgf\" (UniqueName: \"kubernetes.io/projected/1dd94394-046f-4ac1-8cf1-77d7f85d0fe1-kube-api-access-vdrgf\") pod \"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1\" (UID: \"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1\") "
Sep 08 11:56:22.599556 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.599414    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd94394-046f-4ac1-8cf1-77d7f85d0fe1-plugins-config-volume\") pod \"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1\" (UID: \"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1\") "
Sep 08 11:56:22.599556 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.599435    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbtm6\" (UniqueName: \"kubernetes.io/projected/2b801532-302e-465f-ab8f-4ba6abd9211d-kube-api-access-nbtm6\") pod \"2b801532-302e-465f-ab8f-4ba6abd9211d\" (UID: \"2b801532-302e-465f-ab8f-4ba6abd9211d\") "
Sep 08 11:56:22.599556 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.599459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b801532-302e-465f-ab8f-4ba6abd9211d-plugins-config-volume\") pod \"2b801532-302e-465f-ab8f-4ba6abd9211d\" (UID: \"2b801532-302e-465f-ab8f-4ba6abd9211d\") "
Sep 08 11:56:22.599860 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.599837    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd94394-046f-4ac1-8cf1-77d7f85d0fe1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dd94394-046f-4ac1-8cf1-77d7f85d0fe1" (UID: "1dd94394-046f-4ac1-8cf1-77d7f85d0fe1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:56:22.599932 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.599884    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b801532-302e-465f-ab8f-4ba6abd9211d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b801532-302e-465f-ab8f-4ba6abd9211d" (UID: "2b801532-302e-465f-ab8f-4ba6abd9211d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:56:22.601452 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.601433    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b801532-302e-465f-ab8f-4ba6abd9211d-kube-api-access-nbtm6" (OuterVolumeSpecName: "kube-api-access-nbtm6") pod "2b801532-302e-465f-ab8f-4ba6abd9211d" (UID: "2b801532-302e-465f-ab8f-4ba6abd9211d"). InnerVolumeSpecName "kube-api-access-nbtm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:56:22.601522 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.601444    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd94394-046f-4ac1-8cf1-77d7f85d0fe1-kube-api-access-vdrgf" (OuterVolumeSpecName: "kube-api-access-vdrgf") pod "1dd94394-046f-4ac1-8cf1-77d7f85d0fe1" (UID: "1dd94394-046f-4ac1-8cf1-77d7f85d0fe1"). InnerVolumeSpecName "kube-api-access-vdrgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:56:22.700191 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.700160    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdrgf\" (UniqueName: \"kubernetes.io/projected/1dd94394-046f-4ac1-8cf1-77d7f85d0fe1-kube-api-access-vdrgf\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:56:22.700191 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.700189    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd94394-046f-4ac1-8cf1-77d7f85d0fe1-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:56:22.700191 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.700199    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbtm6\" (UniqueName: \"kubernetes.io/projected/2b801532-302e-465f-ab8f-4ba6abd9211d-kube-api-access-nbtm6\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:56:22.700358 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.700207    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b801532-302e-465f-ab8f-4ba6abd9211d-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:56:22.798884 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:22.798800    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:23.430420 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:23.430393    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"
Sep 08 11:56:23.430891 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:23.430419    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79" event={"ID":"2b801532-302e-465f-ab8f-4ba6abd9211d","Type":"ContainerDied","Data":"52b0f819500ab99c693a59e62c669a31df9224dae240b7205dc99192b266b6ed"}
Sep 08 11:56:23.430891 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:23.430508    2561 scope.go:117] "RemoveContainer" containerID="aaec655b619537243eab7c7b09bda3a74b80e85fd3de5776124cd8d574660943"
Sep 08 11:56:23.431827 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:23.431808    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"
Sep 08 11:56:23.431885 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:23.431827    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6" event={"ID":"1dd94394-046f-4ac1-8cf1-77d7f85d0fe1","Type":"ContainerDied","Data":"8ec10748c69506a183d23333b26d980f410e6d6ee66f3ff85d30d4b977930fb9"}
Sep 08 11:56:23.440423 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:23.440404    2561 scope.go:117] "RemoveContainer" containerID="9dc3cfe681b5e50c02cfb3078533c4f9004dfee08404cfbc6f39ef53c22ceeac"
Sep 08 11:56:23.455512 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:23.455489    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"]
Sep 08 11:56:23.462669 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:23.462645    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-64178b-57d668fc55-zvg79"]
Sep 08 11:56:23.475874 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:23.475852    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"]
Sep 08 11:56:23.483413 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:23.483392    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-64178b-7bf9f56fdb-k9bf6"]
Sep 08 11:56:23.961222 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:23.961187    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d9c7f70-045e-4f3e-a040-c9a079e7fdd5" path="/var/lib/kubelet/pods/0d9c7f70-045e-4f3e-a040-c9a079e7fdd5/volumes"
Sep 08 11:56:23.961564 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:23.961551    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd94394-046f-4ac1-8cf1-77d7f85d0fe1" path="/var/lib/kubelet/pods/1dd94394-046f-4ac1-8cf1-77d7f85d0fe1/volumes"
Sep 08 11:56:23.961895 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:23.961882    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b801532-302e-465f-ab8f-4ba6abd9211d" path="/var/lib/kubelet/pods/2b801532-302e-465f-ab8f-4ba6abd9211d/volumes"
Sep 08 11:56:29.239595 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:29.239560    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:29.464617 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:29.464583    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:29.564424 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:29.564340    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:33.238283 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.238242    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb"]
Sep 08 11:56:33.238715 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.238620    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b801532-302e-465f-ab8f-4ba6abd9211d" containerName="payload-processing"
Sep 08 11:56:33.238715 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.238633    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b801532-302e-465f-ab8f-4ba6abd9211d" containerName="payload-processing"
Sep 08 11:56:33.238715 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.238643    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd94394-046f-4ac1-8cf1-77d7f85d0fe1" containerName="payload-pre-processing"
Sep 08 11:56:33.238715 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.238648    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd94394-046f-4ac1-8cf1-77d7f85d0fe1" containerName="payload-pre-processing"
Sep 08 11:56:33.238715 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.238661    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d9c7f70-045e-4f3e-a040-c9a079e7fdd5" containerName="revoke-keys"
Sep 08 11:56:33.238715 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.238667    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d9c7f70-045e-4f3e-a040-c9a079e7fdd5" containerName="revoke-keys"
Sep 08 11:56:33.238901 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.238746    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd94394-046f-4ac1-8cf1-77d7f85d0fe1" containerName="payload-pre-processing"
Sep 08 11:56:33.238901 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.238754    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b801532-302e-465f-ab8f-4ba6abd9211d" containerName="payload-processing"
Sep 08 11:56:33.238901 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.238761    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d9c7f70-045e-4f3e-a040-c9a079e7fdd5" containerName="revoke-keys"
Sep 08 11:56:33.241806 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.241788    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb"
Sep 08 11:56:33.244661 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.244643    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:56:33.244765 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.244659    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:56:33.250611 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.250587    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb"]
Sep 08 11:56:33.355799 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.355765    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:33.392898 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.392870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81965cea-9f34-4034-a674-cc47d3f08b1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb\" (UID: \"81965cea-9f34-4034-a674-cc47d3f08b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb"
Sep 08 11:56:33.493659 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.493583    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81965cea-9f34-4034-a674-cc47d3f08b1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb\" (UID: \"81965cea-9f34-4034-a674-cc47d3f08b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb"
Sep 08 11:56:33.494229 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.494208    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81965cea-9f34-4034-a674-cc47d3f08b1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb\" (UID: \"81965cea-9f34-4034-a674-cc47d3f08b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb"
Sep 08 11:56:33.553624 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.553599    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb"
Sep 08 11:56:33.822921 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:33.822874    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb"]
Sep 08 11:56:33.825164 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:56:33.825135    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81965cea_9f34_4034_a674_cc47d3f08b1f.slice/crio-97250e6f9794f10a4b2ccab5c8e18b59ef7d57bb31376af757a3585571dab6eb WatchSource:0}: Error finding container 97250e6f9794f10a4b2ccab5c8e18b59ef7d57bb31376af757a3585571dab6eb: Status 404 returned error can't find the container with id 97250e6f9794f10a4b2ccab5c8e18b59ef7d57bb31376af757a3585571dab6eb
Sep 08 11:56:34.488370 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:34.488341    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb" event={"ID":"81965cea-9f34-4034-a674-cc47d3f08b1f","Type":"ContainerStarted","Data":"474d9863ef513366afeaa382a079ea7ecb8e9ef639116172269a95819925ddbe"}
Sep 08 11:56:34.488779 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:34.488392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb" event={"ID":"81965cea-9f34-4034-a674-cc47d3f08b1f","Type":"ContainerStarted","Data":"97250e6f9794f10a4b2ccab5c8e18b59ef7d57bb31376af757a3585571dab6eb"}
Sep 08 11:56:34.505802 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:34.505759    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb" podStartSLOduration=1.5057431449999998 podStartE2EDuration="1.505743145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:56:34.505476989 +0000 UTC m=+1601.072237526" watchObservedRunningTime="2026-09-08 11:56:34.505743145 +0000 UTC m=+1601.072503685"
Sep 08 11:56:35.494449 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:35.494414    2561 generic.go:356] "Generic (PLEG): container finished" podID="81965cea-9f34-4034-a674-cc47d3f08b1f" containerID="474d9863ef513366afeaa382a079ea7ecb8e9ef639116172269a95819925ddbe" exitCode=0
Sep 08 11:56:35.494830 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:35.494496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb" event={"ID":"81965cea-9f34-4034-a674-cc47d3f08b1f","Type":"ContainerDied","Data":"474d9863ef513366afeaa382a079ea7ecb8e9ef639116172269a95819925ddbe"}
Sep 08 11:56:36.621853 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:36.621831    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb"
Sep 08 11:56:36.721926 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:36.721884    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81965cea-9f34-4034-a674-cc47d3f08b1f-maas-api-service-ca\") pod \"81965cea-9f34-4034-a674-cc47d3f08b1f\" (UID: \"81965cea-9f34-4034-a674-cc47d3f08b1f\") "
Sep 08 11:56:36.722218 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:36.722195    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81965cea-9f34-4034-a674-cc47d3f08b1f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81965cea-9f34-4034-a674-cc47d3f08b1f" (UID: "81965cea-9f34-4034-a674-cc47d3f08b1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:56:36.823444 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:36.823364    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81965cea-9f34-4034-a674-cc47d3f08b1f-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:56:37.504791 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:37.504761    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb"
Sep 08 11:56:37.504975 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:37.504764    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb" event={"ID":"81965cea-9f34-4034-a674-cc47d3f08b1f","Type":"ContainerDied","Data":"97250e6f9794f10a4b2ccab5c8e18b59ef7d57bb31376af757a3585571dab6eb"}
Sep 08 11:56:37.504975 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:37.504900    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97250e6f9794f10a4b2ccab5c8e18b59ef7d57bb31376af757a3585571dab6eb"
Sep 08 11:56:41.243354 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:41.243324    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:41.345585 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:41.345547    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:43.110127 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:43.110094    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:48.893048 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:48.892966    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:48.933662 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:48.933628    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"]
Sep 08 11:56:48.933888 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:48.933857    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8" podUID="0ad0d1b7-297e-4099-8651-4c3683684140" containerName="payload-pre-processing" containerID="cri-o://f32f642f0f18008633ebe41cc533def35c3f137581e39b0e1c9d8c29f1605c95" gracePeriod=30
Sep 08 11:56:49.044708 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.044669    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:49.129915 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.129878    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb"]
Sep 08 11:56:49.132218 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.132192    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3f9288-967tb"]
Sep 08 11:56:49.135692 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.135665    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:49.186660 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.186638    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"
Sep 08 11:56:49.207213 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.207179    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:49.325445 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.325418    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad0d1b7-297e-4099-8651-4c3683684140-plugins-config-volume\") pod \"0ad0d1b7-297e-4099-8651-4c3683684140\" (UID: \"0ad0d1b7-297e-4099-8651-4c3683684140\") "
Sep 08 11:56:49.325445 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.325447    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntjpr\" (UniqueName: \"kubernetes.io/projected/0ad0d1b7-297e-4099-8651-4c3683684140-kube-api-access-ntjpr\") pod \"0ad0d1b7-297e-4099-8651-4c3683684140\" (UID: \"0ad0d1b7-297e-4099-8651-4c3683684140\") "
Sep 08 11:56:49.325839 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.325811    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad0d1b7-297e-4099-8651-4c3683684140-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ad0d1b7-297e-4099-8651-4c3683684140" (UID: "0ad0d1b7-297e-4099-8651-4c3683684140"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:56:49.327735 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.327702    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad0d1b7-297e-4099-8651-4c3683684140-kube-api-access-ntjpr" (OuterVolumeSpecName: "kube-api-access-ntjpr") pod "0ad0d1b7-297e-4099-8651-4c3683684140" (UID: "0ad0d1b7-297e-4099-8651-4c3683684140"). InnerVolumeSpecName "kube-api-access-ntjpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:56:49.426690 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.426625    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad0d1b7-297e-4099-8651-4c3683684140-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:56:49.426690 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.426648    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntjpr\" (UniqueName: \"kubernetes.io/projected/0ad0d1b7-297e-4099-8651-4c3683684140-kube-api-access-ntjpr\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:56:49.518987 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.518957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:56:49.562128 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.562102    2561 generic.go:356] "Generic (PLEG): container finished" podID="0ad0d1b7-297e-4099-8651-4c3683684140" containerID="f32f642f0f18008633ebe41cc533def35c3f137581e39b0e1c9d8c29f1605c95" exitCode=0
Sep 08 11:56:49.562237 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.562146    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"
Sep 08 11:56:49.562237 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.562189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8" event={"ID":"0ad0d1b7-297e-4099-8651-4c3683684140","Type":"ContainerDied","Data":"f32f642f0f18008633ebe41cc533def35c3f137581e39b0e1c9d8c29f1605c95"}
Sep 08 11:56:49.562319 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.562238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8" event={"ID":"0ad0d1b7-297e-4099-8651-4c3683684140","Type":"ContainerDied","Data":"7654cff4644d1de4e5394c5fabd6d0f76784bf61e82f33a67571910f1184afd9"}
Sep 08 11:56:49.562319 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.562261    2561 scope.go:117] "RemoveContainer" containerID="f32f642f0f18008633ebe41cc533def35c3f137581e39b0e1c9d8c29f1605c95"
Sep 08 11:56:49.571230 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.571215    2561 scope.go:117] "RemoveContainer" containerID="f32f642f0f18008633ebe41cc533def35c3f137581e39b0e1c9d8c29f1605c95"
Sep 08 11:56:49.571528 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:56:49.571509    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32f642f0f18008633ebe41cc533def35c3f137581e39b0e1c9d8c29f1605c95\": container with ID starting with f32f642f0f18008633ebe41cc533def35c3f137581e39b0e1c9d8c29f1605c95 not found: ID does not exist" containerID="f32f642f0f18008633ebe41cc533def35c3f137581e39b0e1c9d8c29f1605c95"
Sep 08 11:56:49.571591 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.571537    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32f642f0f18008633ebe41cc533def35c3f137581e39b0e1c9d8c29f1605c95"} err="failed to get container status \"f32f642f0f18008633ebe41cc533def35c3f137581e39b0e1c9d8c29f1605c95\": rpc error: code = NotFound desc = could not find container \"f32f642f0f18008633ebe41cc533def35c3f137581e39b0e1c9d8c29f1605c95\": container with ID starting with f32f642f0f18008633ebe41cc533def35c3f137581e39b0e1c9d8c29f1605c95 not found: ID does not exist"
Sep 08 11:56:49.585070 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.585041    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"]
Sep 08 11:56:49.589443 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.589422    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f9288-59f79ffcd-4hpt8"]
Sep 08 11:56:49.961398 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.961365    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ad0d1b7-297e-4099-8651-4c3683684140" path="/var/lib/kubelet/pods/0ad0d1b7-297e-4099-8651-4c3683684140/volumes"
Sep 08 11:56:49.961804 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:49.961746    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81965cea-9f34-4034-a674-cc47d3f08b1f" path="/var/lib/kubelet/pods/81965cea-9f34-4034-a674-cc47d3f08b1f/volumes"
Sep 08 11:56:54.336506 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:54.336454    2561 scope.go:117] "RemoveContainer" containerID="e714dc8a6dc25377e0b74c37414af92191afef1501415b7720f599cb68064708"
Sep 08 11:56:54.499813 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:54.499781    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"]
Sep 08 11:56:54.692050 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:56:54.691966    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:09.520308 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.520269    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6" podUID="9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca" containerName="main" containerID="cri-o://4c47b21f8a4aa941bc22054caf0e89986f5ae4ee456698b0540e5f8cda425e43" gracePeriod=45
Sep 08 11:57:09.662162 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.662132    2561 generic.go:356] "Generic (PLEG): container finished" podID="9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca" containerID="4c47b21f8a4aa941bc22054caf0e89986f5ae4ee456698b0540e5f8cda425e43" exitCode=0
Sep 08 11:57:09.662323 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.662205    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6" event={"ID":"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca","Type":"ContainerDied","Data":"4c47b21f8a4aa941bc22054caf0e89986f5ae4ee456698b0540e5f8cda425e43"}
Sep 08 11:57:09.773194 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.773141    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:57:09.783292 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.783274    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-tls-certs\") pod \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") "
Sep 08 11:57:09.783376 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.783303    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-model-cache\") pod \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") "
Sep 08 11:57:09.783376 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.783330    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-dshm\") pod \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") "
Sep 08 11:57:09.783376 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.783359    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-tmp-dir\") pod \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") "
Sep 08 11:57:09.783503 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.783399    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mctgw\" (UniqueName: \"kubernetes.io/projected/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-kube-api-access-mctgw\") pod \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") "
Sep 08 11:57:09.783503 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.783431    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-home\") pod \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\" (UID: \"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca\") "
Sep 08 11:57:09.783612 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.783588    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca" (UID: "9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:57:09.783680 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.783625    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca" (UID: "9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:57:09.783792 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.783768    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-home" (OuterVolumeSpecName: "home") pod "9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca" (UID: "9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:57:09.783846 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.783794    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-model-cache\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:57:09.783846 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.783814    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-tmp-dir\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:57:09.785556 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.785530    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca" (UID: "9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:57:09.785646 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.785617    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-dshm" (OuterVolumeSpecName: "dshm") pod "9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca" (UID: "9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 11:57:09.785646 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.785620    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-kube-api-access-mctgw" (OuterVolumeSpecName: "kube-api-access-mctgw") pod "9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca" (UID: "9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca"). InnerVolumeSpecName "kube-api-access-mctgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:57:09.884184 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.884145    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-dshm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:57:09.884184 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.884178    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mctgw\" (UniqueName: \"kubernetes.io/projected/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-kube-api-access-mctgw\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:57:09.884184 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.884189    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-home\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:57:09.884390 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:09.884198    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca-tls-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:57:10.668206 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:10.668176    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"
Sep 08 11:57:10.668651 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:10.668168    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6" event={"ID":"9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca","Type":"ContainerDied","Data":"b19c58ca91e4a34bff154fd32fb8007552e4d911d26af3179f241e696f88fbf5"}
Sep 08 11:57:10.668651 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:10.668300    2561 scope.go:117] "RemoveContainer" containerID="4c47b21f8a4aa941bc22054caf0e89986f5ae4ee456698b0540e5f8cda425e43"
Sep 08 11:57:10.687567 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:10.687532    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"]
Sep 08 11:57:10.693612 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:10.693587    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-3f9288/ipp-route-3f9288-kserve-5b567ddb95-z88m6"]
Sep 08 11:57:11.960798 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:11.960757    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca" path="/var/lib/kubelet/pods/9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca/volumes"
Sep 08 11:57:22.523683 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:22.523651    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:24.022715 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:24.022681    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:29.026968 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.026920    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:29.090242 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.090207    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:29.310023 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.309943    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:29.692133 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.692049    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm"]
Sep 08 11:57:29.692576 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.692555    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ad0d1b7-297e-4099-8651-4c3683684140" containerName="payload-pre-processing"
Sep 08 11:57:29.692642 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.692580    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ad0d1b7-297e-4099-8651-4c3683684140" containerName="payload-pre-processing"
Sep 08 11:57:29.692642 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.692595    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81965cea-9f34-4034-a674-cc47d3f08b1f" containerName="revoke-keys"
Sep 08 11:57:29.692642 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.692603    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="81965cea-9f34-4034-a674-cc47d3f08b1f" containerName="revoke-keys"
Sep 08 11:57:29.692642 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.692629    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca" containerName="main"
Sep 08 11:57:29.692642 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.692637    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca" containerName="main"
Sep 08 11:57:29.692820 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.692721    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ad0d1b7-297e-4099-8651-4c3683684140" containerName="payload-pre-processing"
Sep 08 11:57:29.692820 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.692736    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aec16f5-ba8c-42bc-b991-e5dc5b3b21ca" containerName="main"
Sep 08 11:57:29.692820 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.692746    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="81965cea-9f34-4034-a674-cc47d3f08b1f" containerName="revoke-keys"
Sep 08 11:57:29.696008 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.695989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm"
Sep 08 11:57:29.698719 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.698690    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:57:29.698830 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.698760    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:57:29.703825 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.703801    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm"]
Sep 08 11:57:29.750939 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.750903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm\" (UID: \"69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm"
Sep 08 11:57:29.851289 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.851254    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm\" (UID: \"69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm"
Sep 08 11:57:29.851944 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.851925    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm\" (UID: \"69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm"
Sep 08 11:57:29.935400 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:29.935363    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:30.006687 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:30.006661    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm"
Sep 08 11:57:30.285306 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:30.285276    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm"]
Sep 08 11:57:30.287413 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:57:30.287389    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d9bfa8_c9f7_42c1_9a71_fb9bfa884ce2.slice/crio-929135de5b7b42676e84accbd6a63e787db80311a704dcb2b7220427690382d1 WatchSource:0}: Error finding container 929135de5b7b42676e84accbd6a63e787db80311a704dcb2b7220427690382d1: Status 404 returned error can't find the container with id 929135de5b7b42676e84accbd6a63e787db80311a704dcb2b7220427690382d1
Sep 08 11:57:30.773391 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:30.773337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm" event={"ID":"69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2","Type":"ContainerStarted","Data":"be6d5ef0ced32e7956822b9e409abb23fbb237f047d7aa19b3f27f5c44809e45"}
Sep 08 11:57:30.773628 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:30.773445    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm" event={"ID":"69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2","Type":"ContainerStarted","Data":"929135de5b7b42676e84accbd6a63e787db80311a704dcb2b7220427690382d1"}
Sep 08 11:57:30.791608 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:30.791561    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm" podStartSLOduration=1.791547542 podStartE2EDuration="1.791547542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:57:30.790505645 +0000 UTC m=+1657.357266236" watchObservedRunningTime="2026-09-08 11:57:30.791547542 +0000 UTC m=+1657.358308082"
Sep 08 11:57:31.779888 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:31.779850    2561 generic.go:356] "Generic (PLEG): container finished" podID="69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2" containerID="be6d5ef0ced32e7956822b9e409abb23fbb237f047d7aa19b3f27f5c44809e45" exitCode=0
Sep 08 11:57:31.780274 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:31.779932    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm" event={"ID":"69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2","Type":"ContainerDied","Data":"be6d5ef0ced32e7956822b9e409abb23fbb237f047d7aa19b3f27f5c44809e45"}
Sep 08 11:57:32.910202 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:32.910182    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm"
Sep 08 11:57:32.979934 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:32.979905    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2-maas-api-service-ca\") pod \"69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2\" (UID: \"69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2\") "
Sep 08 11:57:32.980255 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:32.980231    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2" (UID: "69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:57:33.081229 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:33.081159    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:57:33.410883 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:33.410801    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:33.790984 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:33.790948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm" event={"ID":"69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2","Type":"ContainerDied","Data":"929135de5b7b42676e84accbd6a63e787db80311a704dcb2b7220427690382d1"}
Sep 08 11:57:33.790984 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:33.790979    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm"
Sep 08 11:57:33.791190 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:33.791002    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="929135de5b7b42676e84accbd6a63e787db80311a704dcb2b7220427690382d1"
Sep 08 11:57:40.072264 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:40.072225    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:40.706625 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:40.706587    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:40.735914 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:40.735872    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"]
Sep 08 11:57:40.736157 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:40.736130    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r" podUID="56116325-83f2-4cbf-86d7-3348157096a0" containerName="payload-processing" containerID="cri-o://75d3a419283e00250ed88c1ce7d484dea2b7dae85f9b5e78ced47f923a33747d" gracePeriod=30
Sep 08 11:57:40.806640 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:40.806610    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:40.907327 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:40.907298    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:40.921247 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:40.921221    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm"]
Sep 08 11:57:40.923752 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:40.923727    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2ebeba-89zdm"]
Sep 08 11:57:40.984223 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:40.984203    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"
Sep 08 11:57:41.007949 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.007927    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:41.044019 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.043993    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nfgw\" (UniqueName: \"kubernetes.io/projected/56116325-83f2-4cbf-86d7-3348157096a0-kube-api-access-8nfgw\") pod \"56116325-83f2-4cbf-86d7-3348157096a0\" (UID: \"56116325-83f2-4cbf-86d7-3348157096a0\") "
Sep 08 11:57:41.044138 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.044068    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56116325-83f2-4cbf-86d7-3348157096a0-plugins-config-volume\") pod \"56116325-83f2-4cbf-86d7-3348157096a0\" (UID: \"56116325-83f2-4cbf-86d7-3348157096a0\") "
Sep 08 11:57:41.044527 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.044496    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56116325-83f2-4cbf-86d7-3348157096a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56116325-83f2-4cbf-86d7-3348157096a0" (UID: "56116325-83f2-4cbf-86d7-3348157096a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:57:41.046067 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.046045    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56116325-83f2-4cbf-86d7-3348157096a0-kube-api-access-8nfgw" (OuterVolumeSpecName: "kube-api-access-8nfgw") pod "56116325-83f2-4cbf-86d7-3348157096a0" (UID: "56116325-83f2-4cbf-86d7-3348157096a0"). InnerVolumeSpecName "kube-api-access-8nfgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:57:41.145254 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.145226    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56116325-83f2-4cbf-86d7-3348157096a0-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:57:41.145254 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.145253    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nfgw\" (UniqueName: \"kubernetes.io/projected/56116325-83f2-4cbf-86d7-3348157096a0-kube-api-access-8nfgw\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:57:41.330352 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.330232    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:41.829231 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.829195    2561 generic.go:356] "Generic (PLEG): container finished" podID="56116325-83f2-4cbf-86d7-3348157096a0" containerID="75d3a419283e00250ed88c1ce7d484dea2b7dae85f9b5e78ced47f923a33747d" exitCode=0
Sep 08 11:57:41.829231 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.829235    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"
Sep 08 11:57:41.829507 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.829269    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r" event={"ID":"56116325-83f2-4cbf-86d7-3348157096a0","Type":"ContainerDied","Data":"75d3a419283e00250ed88c1ce7d484dea2b7dae85f9b5e78ced47f923a33747d"}
Sep 08 11:57:41.829507 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.829318    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r" event={"ID":"56116325-83f2-4cbf-86d7-3348157096a0","Type":"ContainerDied","Data":"6e91260a0d17e513c398f11b4cb0072d235d1e4ffac5446fc46fdc37972ec9cc"}
Sep 08 11:57:41.829507 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.829347    2561 scope.go:117] "RemoveContainer" containerID="75d3a419283e00250ed88c1ce7d484dea2b7dae85f9b5e78ced47f923a33747d"
Sep 08 11:57:41.838552 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.838532    2561 scope.go:117] "RemoveContainer" containerID="75d3a419283e00250ed88c1ce7d484dea2b7dae85f9b5e78ced47f923a33747d"
Sep 08 11:57:41.838815 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:57:41.838796    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d3a419283e00250ed88c1ce7d484dea2b7dae85f9b5e78ced47f923a33747d\": container with ID starting with 75d3a419283e00250ed88c1ce7d484dea2b7dae85f9b5e78ced47f923a33747d not found: ID does not exist" containerID="75d3a419283e00250ed88c1ce7d484dea2b7dae85f9b5e78ced47f923a33747d"
Sep 08 11:57:41.838875 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.838822    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d3a419283e00250ed88c1ce7d484dea2b7dae85f9b5e78ced47f923a33747d"} err="failed to get container status \"75d3a419283e00250ed88c1ce7d484dea2b7dae85f9b5e78ced47f923a33747d\": rpc error: code = NotFound desc = could not find container \"75d3a419283e00250ed88c1ce7d484dea2b7dae85f9b5e78ced47f923a33747d\": container with ID starting with 75d3a419283e00250ed88c1ce7d484dea2b7dae85f9b5e78ced47f923a33747d not found: ID does not exist"
Sep 08 11:57:41.851477 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.851450    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"]
Sep 08 11:57:41.856334 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.856315    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2ebeba-5c4c5d674f-chj7r"]
Sep 08 11:57:41.960705 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.960678    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56116325-83f2-4cbf-86d7-3348157096a0" path="/var/lib/kubelet/pods/56116325-83f2-4cbf-86d7-3348157096a0/volumes"
Sep 08 11:57:41.961003 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:41.960990    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2" path="/var/lib/kubelet/pods/69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2/volumes"
Sep 08 11:57:42.522873 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:42.522830    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:42.606638 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:42.606611    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:46.961908 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:46.961872    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:47.064462 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:47.064431    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:48.063402 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:48.063370    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:48.162359 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:48.162327    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:48.390136 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:48.390057    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:48.820178 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:48.820127    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:54.364172 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:54.364139    2561 scope.go:117] "RemoveContainer" containerID="62a7e9620dfa87130f0f2a24cbac0f6656837057571142cd15dacd95ca2d6e69"
Sep 08 11:57:59.340220 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.340178    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:59.360388 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.360353    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"]
Sep 08 11:57:59.360617 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.360591    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b" podUID="62dfa631-bbc0-4811-a0ba-26b7e8b77638" containerName="payload-processing" containerID="cri-o://c99c9be264c4f703889a4d0c5c14be0bb4c069a8ca70cc3f80037f7bfe578191" gracePeriod=30
Sep 08 11:57:59.369671 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.369644    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"]
Sep 08 11:57:59.369857 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.369837    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt" podUID="3cc81383-082f-47f2-90bd-72076fa5bfb3" containerName="payload-pre-processing" containerID="cri-o://433716011e96738bbb96b082fd24ae963f63b769a324a9e3f4108febe1e68924" gracePeriod=30
Sep 08 11:57:59.408403 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.408368    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:59.511944 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.511905    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:59.596682 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.596596    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:59.639845 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.639819    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"
Sep 08 11:57:59.642413 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.642396    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"
Sep 08 11:57:59.798812 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.798773    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62dfa631-bbc0-4811-a0ba-26b7e8b77638-plugins-config-volume\") pod \"62dfa631-bbc0-4811-a0ba-26b7e8b77638\" (UID: \"62dfa631-bbc0-4811-a0ba-26b7e8b77638\") "
Sep 08 11:57:59.798812 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.798811    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb4cr\" (UniqueName: \"kubernetes.io/projected/62dfa631-bbc0-4811-a0ba-26b7e8b77638-kube-api-access-cb4cr\") pod \"62dfa631-bbc0-4811-a0ba-26b7e8b77638\" (UID: \"62dfa631-bbc0-4811-a0ba-26b7e8b77638\") "
Sep 08 11:57:59.799036 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.798831    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpkz5\" (UniqueName: \"kubernetes.io/projected/3cc81383-082f-47f2-90bd-72076fa5bfb3-kube-api-access-xpkz5\") pod \"3cc81383-082f-47f2-90bd-72076fa5bfb3\" (UID: \"3cc81383-082f-47f2-90bd-72076fa5bfb3\") "
Sep 08 11:57:59.799036 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.798882    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc81383-082f-47f2-90bd-72076fa5bfb3-plugins-config-volume\") pod \"3cc81383-082f-47f2-90bd-72076fa5bfb3\" (UID: \"3cc81383-082f-47f2-90bd-72076fa5bfb3\") "
Sep 08 11:57:59.799267 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.799244    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62dfa631-bbc0-4811-a0ba-26b7e8b77638-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62dfa631-bbc0-4811-a0ba-26b7e8b77638" (UID: "62dfa631-bbc0-4811-a0ba-26b7e8b77638"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:57:59.799311 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.799282    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc81383-082f-47f2-90bd-72076fa5bfb3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3cc81383-082f-47f2-90bd-72076fa5bfb3" (UID: "3cc81383-082f-47f2-90bd-72076fa5bfb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:57:59.801041 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.801017    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dfa631-bbc0-4811-a0ba-26b7e8b77638-kube-api-access-cb4cr" (OuterVolumeSpecName: "kube-api-access-cb4cr") pod "62dfa631-bbc0-4811-a0ba-26b7e8b77638" (UID: "62dfa631-bbc0-4811-a0ba-26b7e8b77638"). InnerVolumeSpecName "kube-api-access-cb4cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:57:59.801101 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.801022    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc81383-082f-47f2-90bd-72076fa5bfb3-kube-api-access-xpkz5" (OuterVolumeSpecName: "kube-api-access-xpkz5") pod "3cc81383-082f-47f2-90bd-72076fa5bfb3" (UID: "3cc81383-082f-47f2-90bd-72076fa5bfb3"). InnerVolumeSpecName "kube-api-access-xpkz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:57:59.829782 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.829733    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:59.899875 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.899791    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62dfa631-bbc0-4811-a0ba-26b7e8b77638-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:57:59.899875 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.899822    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb4cr\" (UniqueName: \"kubernetes.io/projected/62dfa631-bbc0-4811-a0ba-26b7e8b77638-kube-api-access-cb4cr\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:57:59.899875 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.899832    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpkz5\" (UniqueName: \"kubernetes.io/projected/3cc81383-082f-47f2-90bd-72076fa5bfb3-kube-api-access-xpkz5\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:57:59.899875 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.899841    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc81383-082f-47f2-90bd-72076fa5bfb3-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:57:59.915173 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.915136    2561 generic.go:356] "Generic (PLEG): container finished" podID="62dfa631-bbc0-4811-a0ba-26b7e8b77638" containerID="c99c9be264c4f703889a4d0c5c14be0bb4c069a8ca70cc3f80037f7bfe578191" exitCode=0
Sep 08 11:57:59.915173 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.915175    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"
Sep 08 11:57:59.915396 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.915228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b" event={"ID":"62dfa631-bbc0-4811-a0ba-26b7e8b77638","Type":"ContainerDied","Data":"c99c9be264c4f703889a4d0c5c14be0bb4c069a8ca70cc3f80037f7bfe578191"}
Sep 08 11:57:59.915396 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.915280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b" event={"ID":"62dfa631-bbc0-4811-a0ba-26b7e8b77638","Type":"ContainerDied","Data":"965b79e99e45cb5e863df7063cb8346bb3ce42ccc46de786993267ec060272da"}
Sep 08 11:57:59.915396 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.915302    2561 scope.go:117] "RemoveContainer" containerID="c99c9be264c4f703889a4d0c5c14be0bb4c069a8ca70cc3f80037f7bfe578191"
Sep 08 11:57:59.916729 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.916705    2561 generic.go:356] "Generic (PLEG): container finished" podID="3cc81383-082f-47f2-90bd-72076fa5bfb3" containerID="433716011e96738bbb96b082fd24ae963f63b769a324a9e3f4108febe1e68924" exitCode=0
Sep 08 11:57:59.916841 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.916744    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"
Sep 08 11:57:59.916841 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.916787    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt" event={"ID":"3cc81383-082f-47f2-90bd-72076fa5bfb3","Type":"ContainerDied","Data":"433716011e96738bbb96b082fd24ae963f63b769a324a9e3f4108febe1e68924"}
Sep 08 11:57:59.916841 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.916829    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt" event={"ID":"3cc81383-082f-47f2-90bd-72076fa5bfb3","Type":"ContainerDied","Data":"1b8bf339a7cd7241772c231daee8cb52e9177b98220d549e1355a0f3252c9624"}
Sep 08 11:57:59.924772 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.924754    2561 scope.go:117] "RemoveContainer" containerID="c99c9be264c4f703889a4d0c5c14be0bb4c069a8ca70cc3f80037f7bfe578191"
Sep 08 11:57:59.925087 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:57:59.925070    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99c9be264c4f703889a4d0c5c14be0bb4c069a8ca70cc3f80037f7bfe578191\": container with ID starting with c99c9be264c4f703889a4d0c5c14be0bb4c069a8ca70cc3f80037f7bfe578191 not found: ID does not exist" containerID="c99c9be264c4f703889a4d0c5c14be0bb4c069a8ca70cc3f80037f7bfe578191"
Sep 08 11:57:59.925179 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.925093    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99c9be264c4f703889a4d0c5c14be0bb4c069a8ca70cc3f80037f7bfe578191"} err="failed to get container status \"c99c9be264c4f703889a4d0c5c14be0bb4c069a8ca70cc3f80037f7bfe578191\": rpc error: code = NotFound desc = could not find container \"c99c9be264c4f703889a4d0c5c14be0bb4c069a8ca70cc3f80037f7bfe578191\": container with ID starting with c99c9be264c4f703889a4d0c5c14be0bb4c069a8ca70cc3f80037f7bfe578191 not found: ID does not exist"
Sep 08 11:57:59.925179 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.925110    2561 scope.go:117] "RemoveContainer" containerID="433716011e96738bbb96b082fd24ae963f63b769a324a9e3f4108febe1e68924"
Sep 08 11:57:59.933224 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.933206    2561 scope.go:117] "RemoveContainer" containerID="433716011e96738bbb96b082fd24ae963f63b769a324a9e3f4108febe1e68924"
Sep 08 11:57:59.933522 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:57:59.933502    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433716011e96738bbb96b082fd24ae963f63b769a324a9e3f4108febe1e68924\": container with ID starting with 433716011e96738bbb96b082fd24ae963f63b769a324a9e3f4108febe1e68924 not found: ID does not exist" containerID="433716011e96738bbb96b082fd24ae963f63b769a324a9e3f4108febe1e68924"
Sep 08 11:57:59.933606 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.933530    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433716011e96738bbb96b082fd24ae963f63b769a324a9e3f4108febe1e68924"} err="failed to get container status \"433716011e96738bbb96b082fd24ae963f63b769a324a9e3f4108febe1e68924\": rpc error: code = NotFound desc = could not find container \"433716011e96738bbb96b082fd24ae963f63b769a324a9e3f4108febe1e68924\": container with ID starting with 433716011e96738bbb96b082fd24ae963f63b769a324a9e3f4108febe1e68924 not found: ID does not exist"
Sep 08 11:57:59.936453 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.936413    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:57:59.945808 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.942462    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"]
Sep 08 11:57:59.948344 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.948318    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-838a68-75b788b7dc-4d59b"]
Sep 08 11:57:59.960612 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.960583    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62dfa631-bbc0-4811-a0ba-26b7e8b77638" path="/var/lib/kubelet/pods/62dfa631-bbc0-4811-a0ba-26b7e8b77638/volumes"
Sep 08 11:57:59.960851 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.960838    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"]
Sep 08 11:57:59.966024 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:57:59.965998    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-838a68-5c9675d8f8-l98lt"]
Sep 08 11:58:01.853866 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:01.853825    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:01.960868 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:01.960833    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cc81383-082f-47f2-90bd-72076fa5bfb3" path="/var/lib/kubelet/pods/3cc81383-082f-47f2-90bd-72076fa5bfb3/volumes"
Sep 08 11:58:07.129521 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.129441    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:07.203335 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.203282    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:07.450430 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.450331    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:07.790564 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.790526    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6"]
Sep 08 11:58:07.790835 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.790823    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56116325-83f2-4cbf-86d7-3348157096a0" containerName="payload-processing"
Sep 08 11:58:07.790879 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.790837    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="56116325-83f2-4cbf-86d7-3348157096a0" containerName="payload-processing"
Sep 08 11:58:07.790879 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.790851    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cc81383-082f-47f2-90bd-72076fa5bfb3" containerName="payload-pre-processing"
Sep 08 11:58:07.790879 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.790857    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cc81383-082f-47f2-90bd-72076fa5bfb3" containerName="payload-pre-processing"
Sep 08 11:58:07.790879 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.790863    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2" containerName="revoke-keys"
Sep 08 11:58:07.790879 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.790869    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2" containerName="revoke-keys"
Sep 08 11:58:07.791055 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.790884    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dfa631-bbc0-4811-a0ba-26b7e8b77638" containerName="payload-processing"
Sep 08 11:58:07.791055 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.790889    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dfa631-bbc0-4811-a0ba-26b7e8b77638" containerName="payload-processing"
Sep 08 11:58:07.791055 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.790940    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="62dfa631-bbc0-4811-a0ba-26b7e8b77638" containerName="payload-processing"
Sep 08 11:58:07.791055 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.790948    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d9bfa8-c9f7-42c1-9a71-fb9bfa884ce2" containerName="revoke-keys"
Sep 08 11:58:07.791055 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.790954    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="56116325-83f2-4cbf-86d7-3348157096a0" containerName="payload-processing"
Sep 08 11:58:07.791055 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.790961    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cc81383-082f-47f2-90bd-72076fa5bfb3" containerName="payload-pre-processing"
Sep 08 11:58:07.799562 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.799519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6"
Sep 08 11:58:07.804488 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.804440    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6"]
Sep 08 11:58:07.805329 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.805306    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 11:58:07.805443 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.805328    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\""
Sep 08 11:58:07.968611 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:07.968575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92796e93-431f-4c0d-ae65-1b8d27ec1010-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6\" (UID: \"92796e93-431f-4c0d-ae65-1b8d27ec1010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6"
Sep 08 11:58:08.069439 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:08.069348    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92796e93-431f-4c0d-ae65-1b8d27ec1010-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6\" (UID: \"92796e93-431f-4c0d-ae65-1b8d27ec1010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6"
Sep 08 11:58:08.070007 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:08.069985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92796e93-431f-4c0d-ae65-1b8d27ec1010-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6\" (UID: \"92796e93-431f-4c0d-ae65-1b8d27ec1010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6"
Sep 08 11:58:08.111732 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:08.111695    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6"
Sep 08 11:58:08.384034 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:08.383969    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6"]
Sep 08 11:58:08.386439 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:58:08.386407    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92796e93_431f_4c0d_ae65_1b8d27ec1010.slice/crio-9526ad15abe645fa45251d516511c3063aa81db74c7748e955b039093f2499f1 WatchSource:0}: Error finding container 9526ad15abe645fa45251d516511c3063aa81db74c7748e955b039093f2499f1: Status 404 returned error can't find the container with id 9526ad15abe645fa45251d516511c3063aa81db74c7748e955b039093f2499f1
Sep 08 11:58:08.961403 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:08.961370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6" event={"ID":"92796e93-431f-4c0d-ae65-1b8d27ec1010","Type":"ContainerStarted","Data":"f82d773688be1676c8a372463ba72bb9ad146f141b2778ce53f9792d834cc77c"}
Sep 08 11:58:08.961600 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:08.961421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6" event={"ID":"92796e93-431f-4c0d-ae65-1b8d27ec1010","Type":"ContainerStarted","Data":"9526ad15abe645fa45251d516511c3063aa81db74c7748e955b039093f2499f1"}
Sep 08 11:58:08.980700 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:08.980536    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6" podStartSLOduration=1.9805221629999998 podStartE2EDuration="1.980522163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:58:08.980265165 +0000 UTC m=+1695.547025707" watchObservedRunningTime="2026-09-08 11:58:08.980522163 +0000 UTC m=+1695.547282705"
Sep 08 11:58:09.967361 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:09.967325    2561 generic.go:356] "Generic (PLEG): container finished" podID="92796e93-431f-4c0d-ae65-1b8d27ec1010" containerID="f82d773688be1676c8a372463ba72bb9ad146f141b2778ce53f9792d834cc77c" exitCode=0
Sep 08 11:58:09.967764 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:09.967408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6" event={"ID":"92796e93-431f-4c0d-ae65-1b8d27ec1010","Type":"ContainerDied","Data":"f82d773688be1676c8a372463ba72bb9ad146f141b2778ce53f9792d834cc77c"}
Sep 08 11:58:11.099234 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:11.099209    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6"
Sep 08 11:58:11.196853 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:11.196815    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92796e93-431f-4c0d-ae65-1b8d27ec1010-maas-api-service-ca\") pod \"92796e93-431f-4c0d-ae65-1b8d27ec1010\" (UID: \"92796e93-431f-4c0d-ae65-1b8d27ec1010\") "
Sep 08 11:58:11.197171 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:11.197147    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92796e93-431f-4c0d-ae65-1b8d27ec1010-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92796e93-431f-4c0d-ae65-1b8d27ec1010" (UID: "92796e93-431f-4c0d-ae65-1b8d27ec1010"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:58:11.298077 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:11.297985    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92796e93-431f-4c0d-ae65-1b8d27ec1010-maas-api-service-ca\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:58:11.977724 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:11.977693    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6"
Sep 08 11:58:11.977893 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:11.977728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6" event={"ID":"92796e93-431f-4c0d-ae65-1b8d27ec1010","Type":"ContainerDied","Data":"9526ad15abe645fa45251d516511c3063aa81db74c7748e955b039093f2499f1"}
Sep 08 11:58:11.977893 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:11.977791    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9526ad15abe645fa45251d516511c3063aa81db74c7748e955b039093f2499f1"
Sep 08 11:58:18.317402 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.317316    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:18.344869 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.344835    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"]
Sep 08 11:58:18.345082 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.345041    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx" podUID="34400d9b-db76-4172-ba9b-b2354f0fcfc2" containerName="payload-processing" containerID="cri-o://a60a7dc80bda451ebec9d8f460416ac0628ac90c626d54e6ed2cc3de63412374" gracePeriod=30
Sep 08 11:58:18.418913 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.418875    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:18.518226 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.518197    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:18.557213 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.557181    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6"]
Sep 08 11:58:18.560328 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.560300    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bb9b64-c9fn6"]
Sep 08 11:58:18.583854 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.583756    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:18.611371 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.611340    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"
Sep 08 11:58:18.653430 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.653395    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7xjh\" (UniqueName: \"kubernetes.io/projected/34400d9b-db76-4172-ba9b-b2354f0fcfc2-kube-api-access-k7xjh\") pod \"34400d9b-db76-4172-ba9b-b2354f0fcfc2\" (UID: \"34400d9b-db76-4172-ba9b-b2354f0fcfc2\") "
Sep 08 11:58:18.653594 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.653462    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34400d9b-db76-4172-ba9b-b2354f0fcfc2-plugins-config-volume\") pod \"34400d9b-db76-4172-ba9b-b2354f0fcfc2\" (UID: \"34400d9b-db76-4172-ba9b-b2354f0fcfc2\") "
Sep 08 11:58:18.653868 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.653841    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34400d9b-db76-4172-ba9b-b2354f0fcfc2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34400d9b-db76-4172-ba9b-b2354f0fcfc2" (UID: "34400d9b-db76-4172-ba9b-b2354f0fcfc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 11:58:18.655504 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.655457    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34400d9b-db76-4172-ba9b-b2354f0fcfc2-kube-api-access-k7xjh" (OuterVolumeSpecName: "kube-api-access-k7xjh") pod "34400d9b-db76-4172-ba9b-b2354f0fcfc2" (UID: "34400d9b-db76-4172-ba9b-b2354f0fcfc2"). InnerVolumeSpecName "kube-api-access-k7xjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:58:18.753929 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.753892    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7xjh\" (UniqueName: \"kubernetes.io/projected/34400d9b-db76-4172-ba9b-b2354f0fcfc2-kube-api-access-k7xjh\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:58:18.753929 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.753923    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34400d9b-db76-4172-ba9b-b2354f0fcfc2-plugins-config-volume\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:58:18.942227 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:18.942140    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:19.014796 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:19.014761    2561 generic.go:356] "Generic (PLEG): container finished" podID="34400d9b-db76-4172-ba9b-b2354f0fcfc2" containerID="a60a7dc80bda451ebec9d8f460416ac0628ac90c626d54e6ed2cc3de63412374" exitCode=0
Sep 08 11:58:19.014971 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:19.014815    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"
Sep 08 11:58:19.014971 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:19.014845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx" event={"ID":"34400d9b-db76-4172-ba9b-b2354f0fcfc2","Type":"ContainerDied","Data":"a60a7dc80bda451ebec9d8f460416ac0628ac90c626d54e6ed2cc3de63412374"}
Sep 08 11:58:19.014971 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:19.014944    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx" event={"ID":"34400d9b-db76-4172-ba9b-b2354f0fcfc2","Type":"ContainerDied","Data":"615f257bd78bc5c133cb22213b845b8cc2382f0d1f7562a80f9fd685aa17fdbb"}
Sep 08 11:58:19.015158 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:19.014971    2561 scope.go:117] "RemoveContainer" containerID="a60a7dc80bda451ebec9d8f460416ac0628ac90c626d54e6ed2cc3de63412374"
Sep 08 11:58:19.024245 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:19.024223    2561 scope.go:117] "RemoveContainer" containerID="a60a7dc80bda451ebec9d8f460416ac0628ac90c626d54e6ed2cc3de63412374"
Sep 08 11:58:19.024664 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:58:19.024645    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60a7dc80bda451ebec9d8f460416ac0628ac90c626d54e6ed2cc3de63412374\": container with ID starting with a60a7dc80bda451ebec9d8f460416ac0628ac90c626d54e6ed2cc3de63412374 not found: ID does not exist" containerID="a60a7dc80bda451ebec9d8f460416ac0628ac90c626d54e6ed2cc3de63412374"
Sep 08 11:58:19.024712 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:19.024673    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60a7dc80bda451ebec9d8f460416ac0628ac90c626d54e6ed2cc3de63412374"} err="failed to get container status \"a60a7dc80bda451ebec9d8f460416ac0628ac90c626d54e6ed2cc3de63412374\": rpc error: code = NotFound desc = could not find container \"a60a7dc80bda451ebec9d8f460416ac0628ac90c626d54e6ed2cc3de63412374\": container with ID starting with a60a7dc80bda451ebec9d8f460416ac0628ac90c626d54e6ed2cc3de63412374 not found: ID does not exist"
Sep 08 11:58:19.039150 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:19.039119    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"]
Sep 08 11:58:19.042229 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:19.042194    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bb9b64-68db9b6699-rwgzx"]
Sep 08 11:58:19.960125 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:19.960089    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34400d9b-db76-4172-ba9b-b2354f0fcfc2" path="/var/lib/kubelet/pods/34400d9b-db76-4172-ba9b-b2354f0fcfc2/volumes"
Sep 08 11:58:19.960524 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:19.960449    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92796e93-431f-4c0d-ae65-1b8d27ec1010" path="/var/lib/kubelet/pods/92796e93-431f-4c0d-ae65-1b8d27ec1010/volumes"
Sep 08 11:58:26.068099 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:26.068061    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:26.197373 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:26.197330    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:26.387185 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:26.387080    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:28.981439 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:28.981399    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-764984dc4c-5jmzf"]
Sep 08 11:58:28.981873 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:28.981662    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-764984dc4c-5jmzf" podUID="293bf1e0-f877-4b8d-b306-ab7d01b0974f" containerName="manager" containerID="cri-o://48223244ec9d38f67fccb4c562f8c94404cb85516991b46d066cb4f70df12866" gracePeriod=30
Sep 08 11:58:29.234952 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:29.234891    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:58:29.334977 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:29.334941    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnn4s\" (UniqueName: \"kubernetes.io/projected/293bf1e0-f877-4b8d-b306-ab7d01b0974f-kube-api-access-hnn4s\") pod \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\" (UID: \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\") "
Sep 08 11:58:29.335156 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:29.335044    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/293bf1e0-f877-4b8d-b306-ab7d01b0974f-metrics-certs\") pod \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\" (UID: \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\") "
Sep 08 11:58:29.335200 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:29.335155    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/293bf1e0-f877-4b8d-b306-ab7d01b0974f-webhook-cert\") pod \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\" (UID: \"293bf1e0-f877-4b8d-b306-ab7d01b0974f\") "
Sep 08 11:58:29.337194 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:29.337162    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/293bf1e0-f877-4b8d-b306-ab7d01b0974f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "293bf1e0-f877-4b8d-b306-ab7d01b0974f" (UID: "293bf1e0-f877-4b8d-b306-ab7d01b0974f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:58:29.337311 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:29.337202    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293bf1e0-f877-4b8d-b306-ab7d01b0974f-kube-api-access-hnn4s" (OuterVolumeSpecName: "kube-api-access-hnn4s") pod "293bf1e0-f877-4b8d-b306-ab7d01b0974f" (UID: "293bf1e0-f877-4b8d-b306-ab7d01b0974f"). InnerVolumeSpecName "kube-api-access-hnn4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:58:29.337311 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:29.337203    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/293bf1e0-f877-4b8d-b306-ab7d01b0974f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "293bf1e0-f877-4b8d-b306-ab7d01b0974f" (UID: "293bf1e0-f877-4b8d-b306-ab7d01b0974f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:58:29.435953 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:29.435907    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/293bf1e0-f877-4b8d-b306-ab7d01b0974f-metrics-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:58:29.435953 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:29.435944    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/293bf1e0-f877-4b8d-b306-ab7d01b0974f-webhook-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:58:29.435953 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:29.435960    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnn4s\" (UniqueName: \"kubernetes.io/projected/293bf1e0-f877-4b8d-b306-ab7d01b0974f-kube-api-access-hnn4s\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:58:30.068167 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:30.068136    2561 generic.go:356] "Generic (PLEG): container finished" podID="293bf1e0-f877-4b8d-b306-ab7d01b0974f" containerID="48223244ec9d38f67fccb4c562f8c94404cb85516991b46d066cb4f70df12866" exitCode=0
Sep 08 11:58:30.068593 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:30.068184    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764984dc4c-5jmzf"
Sep 08 11:58:30.068593 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:30.068216    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764984dc4c-5jmzf" event={"ID":"293bf1e0-f877-4b8d-b306-ab7d01b0974f","Type":"ContainerDied","Data":"48223244ec9d38f67fccb4c562f8c94404cb85516991b46d066cb4f70df12866"}
Sep 08 11:58:30.068593 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:30.068268    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764984dc4c-5jmzf" event={"ID":"293bf1e0-f877-4b8d-b306-ab7d01b0974f","Type":"ContainerDied","Data":"10c79c5963ae4817b4ff4a9020900d7e4e72fc3d023e611e5d40c58710d82cd4"}
Sep 08 11:58:30.068593 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:30.068289    2561 scope.go:117] "RemoveContainer" containerID="48223244ec9d38f67fccb4c562f8c94404cb85516991b46d066cb4f70df12866"
Sep 08 11:58:30.076949 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:30.076931    2561 scope.go:117] "RemoveContainer" containerID="48223244ec9d38f67fccb4c562f8c94404cb85516991b46d066cb4f70df12866"
Sep 08 11:58:30.077229 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:58:30.077211    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48223244ec9d38f67fccb4c562f8c94404cb85516991b46d066cb4f70df12866\": container with ID starting with 48223244ec9d38f67fccb4c562f8c94404cb85516991b46d066cb4f70df12866 not found: ID does not exist" containerID="48223244ec9d38f67fccb4c562f8c94404cb85516991b46d066cb4f70df12866"
Sep 08 11:58:30.077290 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:30.077241    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48223244ec9d38f67fccb4c562f8c94404cb85516991b46d066cb4f70df12866"} err="failed to get container status \"48223244ec9d38f67fccb4c562f8c94404cb85516991b46d066cb4f70df12866\": rpc error: code = NotFound desc = could not find container \"48223244ec9d38f67fccb4c562f8c94404cb85516991b46d066cb4f70df12866\": container with ID starting with 48223244ec9d38f67fccb4c562f8c94404cb85516991b46d066cb4f70df12866 not found: ID does not exist"
Sep 08 11:58:30.088254 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:30.088231    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-764984dc4c-5jmzf"]
Sep 08 11:58:30.093448 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:30.093427    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-764984dc4c-5jmzf"]
Sep 08 11:58:31.964432 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:31.964396    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="293bf1e0-f877-4b8d-b306-ab7d01b0974f" path="/var/lib/kubelet/pods/293bf1e0-f877-4b8d-b306-ab7d01b0974f/volumes"
Sep 08 11:58:43.629142 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:43.629100    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:51.430663 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:51.430628    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:58:51.498162 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:58:51.498131    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:59:09.109539 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:09.109506    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:59:09.177284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:09.177249    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:59:30.937094 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.937031    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-764984dc4c-hp5x9"]
Sep 08 11:59:30.937486 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.937334    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="293bf1e0-f877-4b8d-b306-ab7d01b0974f" containerName="manager"
Sep 08 11:59:30.937486 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.937346    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="293bf1e0-f877-4b8d-b306-ab7d01b0974f" containerName="manager"
Sep 08 11:59:30.937486 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.937358    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34400d9b-db76-4172-ba9b-b2354f0fcfc2" containerName="payload-processing"
Sep 08 11:59:30.937486 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.937367    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="34400d9b-db76-4172-ba9b-b2354f0fcfc2" containerName="payload-processing"
Sep 08 11:59:30.937486 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.937376    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92796e93-431f-4c0d-ae65-1b8d27ec1010" containerName="revoke-keys"
Sep 08 11:59:30.937486 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.937381    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="92796e93-431f-4c0d-ae65-1b8d27ec1010" containerName="revoke-keys"
Sep 08 11:59:30.937486 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.937445    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="293bf1e0-f877-4b8d-b306-ab7d01b0974f" containerName="manager"
Sep 08 11:59:30.937486 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.937453    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="92796e93-431f-4c0d-ae65-1b8d27ec1010" containerName="revoke-keys"
Sep 08 11:59:30.937486 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.937462    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="34400d9b-db76-4172-ba9b-b2354f0fcfc2" containerName="payload-processing"
Sep 08 11:59:30.940262 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.940246    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:30.943284 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.943253    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 11:59:30.944478 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.944443    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 11:59:30.944478 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.944459    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-z8t97\""
Sep 08 11:59:30.944653 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.944495    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 11:59:30.957699 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:30.957669    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-764984dc4c-hp5x9"]
Sep 08 11:59:31.044434 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:31.044390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc2f031d-acf5-4206-a403-3b9f7bdae263-metrics-certs\") pod \"maas-controller-764984dc4c-hp5x9\" (UID: \"fc2f031d-acf5-4206-a403-3b9f7bdae263\") " pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:31.044434 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:31.044432    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2f031d-acf5-4206-a403-3b9f7bdae263-webhook-cert\") pod \"maas-controller-764984dc4c-hp5x9\" (UID: \"fc2f031d-acf5-4206-a403-3b9f7bdae263\") " pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:31.044677 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:31.044609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wzts\" (UniqueName: \"kubernetes.io/projected/fc2f031d-acf5-4206-a403-3b9f7bdae263-kube-api-access-2wzts\") pod \"maas-controller-764984dc4c-hp5x9\" (UID: \"fc2f031d-acf5-4206-a403-3b9f7bdae263\") " pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:31.145781 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:31.145745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wzts\" (UniqueName: \"kubernetes.io/projected/fc2f031d-acf5-4206-a403-3b9f7bdae263-kube-api-access-2wzts\") pod \"maas-controller-764984dc4c-hp5x9\" (UID: \"fc2f031d-acf5-4206-a403-3b9f7bdae263\") " pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:31.145942 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:31.145815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc2f031d-acf5-4206-a403-3b9f7bdae263-metrics-certs\") pod \"maas-controller-764984dc4c-hp5x9\" (UID: \"fc2f031d-acf5-4206-a403-3b9f7bdae263\") " pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:31.145942 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:31.145851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2f031d-acf5-4206-a403-3b9f7bdae263-webhook-cert\") pod \"maas-controller-764984dc4c-hp5x9\" (UID: \"fc2f031d-acf5-4206-a403-3b9f7bdae263\") " pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:31.148347 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:31.148318    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc2f031d-acf5-4206-a403-3b9f7bdae263-metrics-certs\") pod \"maas-controller-764984dc4c-hp5x9\" (UID: \"fc2f031d-acf5-4206-a403-3b9f7bdae263\") " pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:31.148458 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:31.148373    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2f031d-acf5-4206-a403-3b9f7bdae263-webhook-cert\") pod \"maas-controller-764984dc4c-hp5x9\" (UID: \"fc2f031d-acf5-4206-a403-3b9f7bdae263\") " pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:31.156380 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:31.156355    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wzts\" (UniqueName: \"kubernetes.io/projected/fc2f031d-acf5-4206-a403-3b9f7bdae263-kube-api-access-2wzts\") pod \"maas-controller-764984dc4c-hp5x9\" (UID: \"fc2f031d-acf5-4206-a403-3b9f7bdae263\") " pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:31.250859 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:31.250819    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:31.523540 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:31.523512    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-764984dc4c-hp5x9"]
Sep 08 11:59:31.525943 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:59:31.525916    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2f031d_acf5_4206_a403_3b9f7bdae263.slice/crio-a8dce79e18db4aa8b26fc57cc63063b370831f094c37b6ef769e58fbd8a6947c WatchSource:0}: Error finding container a8dce79e18db4aa8b26fc57cc63063b370831f094c37b6ef769e58fbd8a6947c: Status 404 returned error can't find the container with id a8dce79e18db4aa8b26fc57cc63063b370831f094c37b6ef769e58fbd8a6947c
Sep 08 11:59:31.527247 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:31.527226    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 11:59:32.346438 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:32.346402    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764984dc4c-hp5x9" event={"ID":"fc2f031d-acf5-4206-a403-3b9f7bdae263","Type":"ContainerStarted","Data":"07c1fbf14ef78536372266b8e732e5a5434376187c58f2eb0da49cfa87a79909"}
Sep 08 11:59:32.346876 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:32.346475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764984dc4c-hp5x9" event={"ID":"fc2f031d-acf5-4206-a403-3b9f7bdae263","Type":"ContainerStarted","Data":"a8dce79e18db4aa8b26fc57cc63063b370831f094c37b6ef769e58fbd8a6947c"}
Sep 08 11:59:32.346876 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:32.346597    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:32.368857 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:32.368803    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-764984dc4c-hp5x9" podStartSLOduration=1.657760111 podStartE2EDuration="2.368786222s" totalImagesPullingTime="711.026111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:59:32.367031873 +0000 UTC m=+1778.933792414" watchObservedRunningTime="2026-09-08 11:59:32.368786222 +0000 UTC m=+1778.935546763"
Sep 08 11:59:43.353128 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:43.353096    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:50.382107 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:50.382022    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:59:50.637099 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:50.636982    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:59:50.728814 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:50.728774    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-764984dc4c-hp5x9"]
Sep 08 11:59:50.729028 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:50.728989    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-764984dc4c-hp5x9" podUID="fc2f031d-acf5-4206-a403-3b9f7bdae263" containerName="manager" containerID="cri-o://07c1fbf14ef78536372266b8e732e5a5434376187c58f2eb0da49cfa87a79909" gracePeriod=30
Sep 08 11:59:50.969070 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:50.969043    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:51.118459 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.118419    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wzts\" (UniqueName: \"kubernetes.io/projected/fc2f031d-acf5-4206-a403-3b9f7bdae263-kube-api-access-2wzts\") pod \"fc2f031d-acf5-4206-a403-3b9f7bdae263\" (UID: \"fc2f031d-acf5-4206-a403-3b9f7bdae263\") "
Sep 08 11:59:51.118684 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.118531    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc2f031d-acf5-4206-a403-3b9f7bdae263-metrics-certs\") pod \"fc2f031d-acf5-4206-a403-3b9f7bdae263\" (UID: \"fc2f031d-acf5-4206-a403-3b9f7bdae263\") "
Sep 08 11:59:51.118684 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.118553    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2f031d-acf5-4206-a403-3b9f7bdae263-webhook-cert\") pod \"fc2f031d-acf5-4206-a403-3b9f7bdae263\" (UID: \"fc2f031d-acf5-4206-a403-3b9f7bdae263\") "
Sep 08 11:59:51.120648 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.120614    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2f031d-acf5-4206-a403-3b9f7bdae263-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fc2f031d-acf5-4206-a403-3b9f7bdae263" (UID: "fc2f031d-acf5-4206-a403-3b9f7bdae263"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:59:51.120648 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.120630    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2f031d-acf5-4206-a403-3b9f7bdae263-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fc2f031d-acf5-4206-a403-3b9f7bdae263" (UID: "fc2f031d-acf5-4206-a403-3b9f7bdae263"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 11:59:51.120791 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.120630    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2f031d-acf5-4206-a403-3b9f7bdae263-kube-api-access-2wzts" (OuterVolumeSpecName: "kube-api-access-2wzts") pod "fc2f031d-acf5-4206-a403-3b9f7bdae263" (UID: "fc2f031d-acf5-4206-a403-3b9f7bdae263"). InnerVolumeSpecName "kube-api-access-2wzts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 11:59:51.219322 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.219289    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc2f031d-acf5-4206-a403-3b9f7bdae263-metrics-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:59:51.219322 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.219318    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc2f031d-acf5-4206-a403-3b9f7bdae263-webhook-cert\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:59:51.219322 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.219327    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wzts\" (UniqueName: \"kubernetes.io/projected/fc2f031d-acf5-4206-a403-3b9f7bdae263-kube-api-access-2wzts\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 11:59:51.433295 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.433259    2561 generic.go:356] "Generic (PLEG): container finished" podID="fc2f031d-acf5-4206-a403-3b9f7bdae263" containerID="07c1fbf14ef78536372266b8e732e5a5434376187c58f2eb0da49cfa87a79909" exitCode=0
Sep 08 11:59:51.433797 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.433307    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764984dc4c-hp5x9"
Sep 08 11:59:51.433797 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.433338    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764984dc4c-hp5x9" event={"ID":"fc2f031d-acf5-4206-a403-3b9f7bdae263","Type":"ContainerDied","Data":"07c1fbf14ef78536372266b8e732e5a5434376187c58f2eb0da49cfa87a79909"}
Sep 08 11:59:51.433797 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.433390    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764984dc4c-hp5x9" event={"ID":"fc2f031d-acf5-4206-a403-3b9f7bdae263","Type":"ContainerDied","Data":"a8dce79e18db4aa8b26fc57cc63063b370831f094c37b6ef769e58fbd8a6947c"}
Sep 08 11:59:51.433797 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.433411    2561 scope.go:117] "RemoveContainer" containerID="07c1fbf14ef78536372266b8e732e5a5434376187c58f2eb0da49cfa87a79909"
Sep 08 11:59:51.442036 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.442018    2561 scope.go:117] "RemoveContainer" containerID="07c1fbf14ef78536372266b8e732e5a5434376187c58f2eb0da49cfa87a79909"
Sep 08 11:59:51.442308 ip-10-0-139-41 kubenswrapper[2561]: E0908 11:59:51.442289    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c1fbf14ef78536372266b8e732e5a5434376187c58f2eb0da49cfa87a79909\": container with ID starting with 07c1fbf14ef78536372266b8e732e5a5434376187c58f2eb0da49cfa87a79909 not found: ID does not exist" containerID="07c1fbf14ef78536372266b8e732e5a5434376187c58f2eb0da49cfa87a79909"
Sep 08 11:59:51.442374 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.442320    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c1fbf14ef78536372266b8e732e5a5434376187c58f2eb0da49cfa87a79909"} err="failed to get container status \"07c1fbf14ef78536372266b8e732e5a5434376187c58f2eb0da49cfa87a79909\": rpc error: code = NotFound desc = could not find container \"07c1fbf14ef78536372266b8e732e5a5434376187c58f2eb0da49cfa87a79909\": container with ID starting with 07c1fbf14ef78536372266b8e732e5a5434376187c58f2eb0da49cfa87a79909 not found: ID does not exist"
Sep 08 11:59:51.457232 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.457197    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-764984dc4c-hp5x9"]
Sep 08 11:59:51.464310 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.464278    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-764984dc4c-hp5x9"]
Sep 08 11:59:51.960600 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:51.960566    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc2f031d-acf5-4206-a403-3b9f7bdae263" path="/var/lib/kubelet/pods/fc2f031d-acf5-4206-a403-3b9f7bdae263/volumes"
Sep 08 11:59:52.160668 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.160641    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 11:59:52.261222 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.261144    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-764984dc4c-ppn5r"]
Sep 08 11:59:52.261456 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.261443    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2f031d-acf5-4206-a403-3b9f7bdae263" containerName="manager"
Sep 08 11:59:52.261521 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.261457    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2f031d-acf5-4206-a403-3b9f7bdae263" containerName="manager"
Sep 08 11:59:52.261558 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.261540    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2f031d-acf5-4206-a403-3b9f7bdae263" containerName="manager"
Sep 08 11:59:52.264421 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.264404    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764984dc4c-ppn5r"
Sep 08 11:59:52.268432 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.268412    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 11:59:52.268691 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.268672    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-z8t97\""
Sep 08 11:59:52.268824 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.268677    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 11:59:52.268824 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.268691    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 11:59:52.274581 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.274556    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-764984dc4c-ppn5r"]
Sep 08 11:59:52.327839 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.327802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a7207aa-fe29-4074-9d1c-edefc51911e5-webhook-cert\") pod \"maas-controller-764984dc4c-ppn5r\" (UID: \"7a7207aa-fe29-4074-9d1c-edefc51911e5\") " pod="opendatahub/maas-controller-764984dc4c-ppn5r"
Sep 08 11:59:52.327839 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.327847    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7207aa-fe29-4074-9d1c-edefc51911e5-metrics-certs\") pod \"maas-controller-764984dc4c-ppn5r\" (UID: \"7a7207aa-fe29-4074-9d1c-edefc51911e5\") " pod="opendatahub/maas-controller-764984dc4c-ppn5r"
Sep 08 11:59:52.328099 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.327864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7bwv\" (UniqueName: \"kubernetes.io/projected/7a7207aa-fe29-4074-9d1c-edefc51911e5-kube-api-access-p7bwv\") pod \"maas-controller-764984dc4c-ppn5r\" (UID: \"7a7207aa-fe29-4074-9d1c-edefc51911e5\") " pod="opendatahub/maas-controller-764984dc4c-ppn5r"
Sep 08 11:59:52.429049 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.429005    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a7207aa-fe29-4074-9d1c-edefc51911e5-webhook-cert\") pod \"maas-controller-764984dc4c-ppn5r\" (UID: \"7a7207aa-fe29-4074-9d1c-edefc51911e5\") " pod="opendatahub/maas-controller-764984dc4c-ppn5r"
Sep 08 11:59:52.429218 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.429064    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7207aa-fe29-4074-9d1c-edefc51911e5-metrics-certs\") pod \"maas-controller-764984dc4c-ppn5r\" (UID: \"7a7207aa-fe29-4074-9d1c-edefc51911e5\") " pod="opendatahub/maas-controller-764984dc4c-ppn5r"
Sep 08 11:59:52.429218 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.429086    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7bwv\" (UniqueName: \"kubernetes.io/projected/7a7207aa-fe29-4074-9d1c-edefc51911e5-kube-api-access-p7bwv\") pod \"maas-controller-764984dc4c-ppn5r\" (UID: \"7a7207aa-fe29-4074-9d1c-edefc51911e5\") " pod="opendatahub/maas-controller-764984dc4c-ppn5r"
Sep 08 11:59:52.431558 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.431534    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7207aa-fe29-4074-9d1c-edefc51911e5-metrics-certs\") pod \"maas-controller-764984dc4c-ppn5r\" (UID: \"7a7207aa-fe29-4074-9d1c-edefc51911e5\") " pod="opendatahub/maas-controller-764984dc4c-ppn5r"
Sep 08 11:59:52.431676 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.431643    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a7207aa-fe29-4074-9d1c-edefc51911e5-webhook-cert\") pod \"maas-controller-764984dc4c-ppn5r\" (UID: \"7a7207aa-fe29-4074-9d1c-edefc51911e5\") " pod="opendatahub/maas-controller-764984dc4c-ppn5r"
Sep 08 11:59:52.438954 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.438929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7bwv\" (UniqueName: \"kubernetes.io/projected/7a7207aa-fe29-4074-9d1c-edefc51911e5-kube-api-access-p7bwv\") pod \"maas-controller-764984dc4c-ppn5r\" (UID: \"7a7207aa-fe29-4074-9d1c-edefc51911e5\") " pod="opendatahub/maas-controller-764984dc4c-ppn5r"
Sep 08 11:59:52.575617 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.575504    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-764984dc4c-ppn5r"
Sep 08 11:59:52.855490 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:52.853687    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-764984dc4c-ppn5r"]
Sep 08 11:59:52.858030 ip-10-0-139-41 kubenswrapper[2561]: W0908 11:59:52.857997    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7207aa_fe29_4074_9d1c_edefc51911e5.slice/crio-3c67347dfe2710723402760587843adbb8d10ad2855b1c90bfbbf76f0e0a6d2a WatchSource:0}: Error finding container 3c67347dfe2710723402760587843adbb8d10ad2855b1c90bfbbf76f0e0a6d2a: Status 404 returned error can't find the container with id 3c67347dfe2710723402760587843adbb8d10ad2855b1c90bfbbf76f0e0a6d2a
Sep 08 11:59:53.445813 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:53.445774    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764984dc4c-ppn5r" event={"ID":"7a7207aa-fe29-4074-9d1c-edefc51911e5","Type":"ContainerStarted","Data":"3c67347dfe2710723402760587843adbb8d10ad2855b1c90bfbbf76f0e0a6d2a"}
Sep 08 11:59:54.451851 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:54.451812    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-764984dc4c-ppn5r" event={"ID":"7a7207aa-fe29-4074-9d1c-edefc51911e5","Type":"ContainerStarted","Data":"8aee16222b69bfc85cadd4d92ea8389617cf6498740dafdf9e58b6d80e36249a"}
Sep 08 11:59:54.452267 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:54.451961    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-764984dc4c-ppn5r"
Sep 08 11:59:54.472232 ip-10-0-139-41 kubenswrapper[2561]: I0908 11:59:54.472178    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-764984dc4c-ppn5r" podStartSLOduration=1.636179203 podStartE2EDuration="2.472160627s" totalImagesPullingTime="835.981424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:59:54.471046846 +0000 UTC m=+1801.037807399" watchObservedRunningTime="2026-09-08 11:59:54.472160627 +0000 UTC m=+1801.038921168"
Sep 08 12:00:05.459356 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:00:05.459325    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-764984dc4c-ppn5r"
Sep 08 12:00:54.418741 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:00:54.418710    2561 scope.go:117] "RemoveContainer" containerID="433c9758343e30749b8d73b98fa649a13253dc90b21549076fcb5e4fca8b922d"
Sep 08 12:01:54.438215 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:01:54.438178    2561 scope.go:117] "RemoveContainer" containerID="c9b7ee24a526d45fa93b9451e16cfd493b674fa542458f61fcac4ba1f8f77c62"
Sep 08 12:01:54.446208 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:01:54.446185    2561 scope.go:117] "RemoveContainer" containerID="59e4294fcb9c500a1a12a371e4c52ddb6b3e7e711d943b5e30e3019b0c1cfb11"
Sep 08 12:01:54.453612 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:01:54.453593    2561 scope.go:117] "RemoveContainer" containerID="afe2134ee190c357cc910d51078c813260fc5c0d2eced9c91ca64cd8caab5fb1"
Sep 08 12:02:48.049528 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.049441    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"]
Sep 08 12:02:48.055128 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.055096    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.058407 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.058381    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9d73e940-kserve-self-signed-certs\""
Sep 08 12:02:48.065293 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.065265    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"]
Sep 08 12:02:48.137034 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.136998    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 12:02:48.215228 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.215189    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-tmp-dir\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.215228 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.215225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c21df69-1c10-4ee2-aa0d-05764e87208c-tls-certs\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.215454 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.215264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-dshm\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.215454 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.215302    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-home\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.215454 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.215321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-model-cache\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.215454 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.215383    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf9sx\" (UniqueName: \"kubernetes.io/projected/4c21df69-1c10-4ee2-aa0d-05764e87208c-kube-api-access-lf9sx\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.316399 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.316299    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9sx\" (UniqueName: \"kubernetes.io/projected/4c21df69-1c10-4ee2-aa0d-05764e87208c-kube-api-access-lf9sx\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.316399 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.316359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-tmp-dir\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.316399 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.316377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c21df69-1c10-4ee2-aa0d-05764e87208c-tls-certs\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.316719 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.316414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-dshm\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.316719 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.316434    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-home\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.316719 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.316451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-model-cache\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.316877 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.316853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-model-cache\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.316941 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.316921    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-home\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.317002 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.316952    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-tmp-dir\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.318822 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.318799    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-dshm\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.318990 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.318972    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c21df69-1c10-4ee2-aa0d-05764e87208c-tls-certs\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.326078 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.326056    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf9sx\" (UniqueName: \"kubernetes.io/projected/4c21df69-1c10-4ee2-aa0d-05764e87208c-kube-api-access-lf9sx\") pod \"e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") " pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.364981 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.364931    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:48.637651 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:48.637559    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"]
Sep 08 12:02:48.640517 ip-10-0-139-41 kubenswrapper[2561]: W0908 12:02:48.640487    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c21df69_1c10_4ee2_aa0d_05764e87208c.slice/crio-a8ff5b0ed509cce318f6f6d7bd0c4210e2158961f2109a86a2f01bce6cab3d3c WatchSource:0}: Error finding container a8ff5b0ed509cce318f6f6d7bd0c4210e2158961f2109a86a2f01bce6cab3d3c: Status 404 returned error can't find the container with id a8ff5b0ed509cce318f6f6d7bd0c4210e2158961f2109a86a2f01bce6cab3d3c
Sep 08 12:02:49.247148 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:49.247106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx" event={"ID":"4c21df69-1c10-4ee2-aa0d-05764e87208c","Type":"ContainerStarted","Data":"9beba3478b7a4089fbe4a41308a21f074819b8ce8d68fa8f6c72d38daaee583a"}
Sep 08 12:02:49.247556 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:49.247156    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx" event={"ID":"4c21df69-1c10-4ee2-aa0d-05764e87208c","Type":"ContainerStarted","Data":"a8ff5b0ed509cce318f6f6d7bd0c4210e2158961f2109a86a2f01bce6cab3d3c"}
Sep 08 12:02:49.267954 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:49.267904    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx" podStartSLOduration=1.267887955 podStartE2EDuration="1.267887955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 12:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 12:02:49.267268952 +0000 UTC m=+1975.834029493" watchObservedRunningTime="2026-09-08 12:02:49.267887955 +0000 UTC m=+1975.834648495"
Sep 08 12:02:54.476723 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:54.476689    2561 scope.go:117] "RemoveContainer" containerID="474d9863ef513366afeaa382a079ea7ecb8e9ef639116172269a95819925ddbe"
Sep 08 12:02:54.484553 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:54.484533    2561 scope.go:117] "RemoveContainer" containerID="178a25d65214a70aaafa47b641b568146a401930f2e4e3dfb5e1a7bdb1f8bed2"
Sep 08 12:02:58.365280 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:58.365241    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:58.365706 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:58.365295    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:58.377654 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:58.377622    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:02:59.304597 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:02:59.304561    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:03:09.523624 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.523583    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"]
Sep 08 12:03:09.526782 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.526763    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.533774 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.533754    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-9d73e940-kserve-self-signed-certs\""
Sep 08 12:03:09.541104 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.541071    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"]
Sep 08 12:03:09.586604 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.586561    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-tmp-dir\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.586780 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.586611    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bltd9\" (UniqueName: \"kubernetes.io/projected/d66f5116-f1c6-4b58-a569-298da915fd7a-kube-api-access-bltd9\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.586780 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.586635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-home\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.586780 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.586698    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66f5116-f1c6-4b58-a569-298da915fd7a-tls-certs\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.586780 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.586757    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-model-cache\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.586780 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.586774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-dshm\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.687628 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.687585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bltd9\" (UniqueName: \"kubernetes.io/projected/d66f5116-f1c6-4b58-a569-298da915fd7a-kube-api-access-bltd9\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.687803 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.687636    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-home\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.687803 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.687680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66f5116-f1c6-4b58-a569-298da915fd7a-tls-certs\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.687803 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.687722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-model-cache\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.687803 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.687742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-dshm\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.687803 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.687776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-tmp-dir\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.688126 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.688089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-home\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.688196 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.688135    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-tmp-dir\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.688196 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.688113    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-model-cache\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.690087 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.690070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-dshm\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.690354 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.690336    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66f5116-f1c6-4b58-a569-298da915fd7a-tls-certs\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.697364 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.697336    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 12:03:09.700122 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.700091    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bltd9\" (UniqueName: \"kubernetes.io/projected/d66f5116-f1c6-4b58-a569-298da915fd7a-kube-api-access-bltd9\") pod \"e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") " pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:09.836244 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:09.836133    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:10.115002 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:10.114974    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"]
Sep 08 12:03:10.117201 ip-10-0-139-41 kubenswrapper[2561]: W0908 12:03:10.117166    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd66f5116_f1c6_4b58_a569_298da915fd7a.slice/crio-155078a00c8189ad36e627537d63e21ad6da0267f022d1693eb4592223c2e03e WatchSource:0}: Error finding container 155078a00c8189ad36e627537d63e21ad6da0267f022d1693eb4592223c2e03e: Status 404 returned error can't find the container with id 155078a00c8189ad36e627537d63e21ad6da0267f022d1693eb4592223c2e03e
Sep 08 12:03:10.337586 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:10.337546    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg" event={"ID":"d66f5116-f1c6-4b58-a569-298da915fd7a","Type":"ContainerStarted","Data":"f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757"}
Sep 08 12:03:10.337780 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:10.337597    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg" event={"ID":"d66f5116-f1c6-4b58-a569-298da915fd7a","Type":"ContainerStarted","Data":"155078a00c8189ad36e627537d63e21ad6da0267f022d1693eb4592223c2e03e"}
Sep 08 12:03:10.358789 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:10.358723    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg" podStartSLOduration=1.358705098 podStartE2EDuration="1.358705098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 12:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 12:03:10.358013133 +0000 UTC m=+1996.924773675" watchObservedRunningTime="2026-09-08 12:03:10.358705098 +0000 UTC m=+1996.925465639"
Sep 08 12:03:19.836916 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:19.836861    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:19.836916 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:19.836928    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:19.850382 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:19.850348    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:20.394619 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:20.394587    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:31.487877 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:31.487848    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 12:03:42.610423 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:42.610388    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"]
Sep 08 12:03:42.640395 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:42.640367    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 12:03:44.547035 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:44.546993    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"]
Sep 08 12:03:44.567762 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:44.567733    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 12:03:54.505247 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:54.505214    2561 scope.go:117] "RemoveContainer" containerID="be6d5ef0ced32e7956822b9e409abb23fbb237f047d7aa19b3f27f5c44809e45"
Sep 08 12:03:57.629537 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.629459    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx" podUID="4c21df69-1c10-4ee2-aa0d-05764e87208c" containerName="main" containerID="cri-o://9beba3478b7a4089fbe4a41308a21f074819b8ce8d68fa8f6c72d38daaee583a" gracePeriod=45
Sep 08 12:03:57.873511 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.873488    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:03:57.893204 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.893145    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-dshm\") pod \"4c21df69-1c10-4ee2-aa0d-05764e87208c\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") "
Sep 08 12:03:57.893204 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.893173    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-model-cache\") pod \"4c21df69-1c10-4ee2-aa0d-05764e87208c\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") "
Sep 08 12:03:57.893204 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.893191    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf9sx\" (UniqueName: \"kubernetes.io/projected/4c21df69-1c10-4ee2-aa0d-05764e87208c-kube-api-access-lf9sx\") pod \"4c21df69-1c10-4ee2-aa0d-05764e87208c\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") "
Sep 08 12:03:57.893442 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.893228    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-home\") pod \"4c21df69-1c10-4ee2-aa0d-05764e87208c\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") "
Sep 08 12:03:57.893442 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.893257    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-tmp-dir\") pod \"4c21df69-1c10-4ee2-aa0d-05764e87208c\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") "
Sep 08 12:03:57.893442 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.893287    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c21df69-1c10-4ee2-aa0d-05764e87208c-tls-certs\") pod \"4c21df69-1c10-4ee2-aa0d-05764e87208c\" (UID: \"4c21df69-1c10-4ee2-aa0d-05764e87208c\") "
Sep 08 12:03:57.893615 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.893453    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c21df69-1c10-4ee2-aa0d-05764e87208c" (UID: "4c21df69-1c10-4ee2-aa0d-05764e87208c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 12:03:57.893615 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.893484    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-home" (OuterVolumeSpecName: "home") pod "4c21df69-1c10-4ee2-aa0d-05764e87208c" (UID: "4c21df69-1c10-4ee2-aa0d-05764e87208c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 12:03:57.893615 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.893572    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c21df69-1c10-4ee2-aa0d-05764e87208c" (UID: "4c21df69-1c10-4ee2-aa0d-05764e87208c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 12:03:57.895206 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.895174    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-dshm" (OuterVolumeSpecName: "dshm") pod "4c21df69-1c10-4ee2-aa0d-05764e87208c" (UID: "4c21df69-1c10-4ee2-aa0d-05764e87208c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 12:03:57.895401 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.895379    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c21df69-1c10-4ee2-aa0d-05764e87208c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c21df69-1c10-4ee2-aa0d-05764e87208c" (UID: "4c21df69-1c10-4ee2-aa0d-05764e87208c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 12:03:57.895514 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.895401    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c21df69-1c10-4ee2-aa0d-05764e87208c-kube-api-access-lf9sx" (OuterVolumeSpecName: "kube-api-access-lf9sx") pod "4c21df69-1c10-4ee2-aa0d-05764e87208c" (UID: "4c21df69-1c10-4ee2-aa0d-05764e87208c"). InnerVolumeSpecName "kube-api-access-lf9sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 12:03:57.994114 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.994082    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-dshm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 12:03:57.994114 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.994108    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-model-cache\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 12:03:57.994114 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.994119    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf9sx\" (UniqueName: \"kubernetes.io/projected/4c21df69-1c10-4ee2-aa0d-05764e87208c-kube-api-access-lf9sx\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 12:03:57.994322 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.994128    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-home\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 12:03:57.994322 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.994136    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c21df69-1c10-4ee2-aa0d-05764e87208c-tmp-dir\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 12:03:57.994322 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:57.994144    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c21df69-1c10-4ee2-aa0d-05764e87208c-tls-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 12:03:58.548946 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:58.548911    2561 generic.go:356] "Generic (PLEG): container finished" podID="4c21df69-1c10-4ee2-aa0d-05764e87208c" containerID="9beba3478b7a4089fbe4a41308a21f074819b8ce8d68fa8f6c72d38daaee583a" exitCode=0
Sep 08 12:03:58.549152 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:58.548956    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"
Sep 08 12:03:58.549152 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:58.549002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx" event={"ID":"4c21df69-1c10-4ee2-aa0d-05764e87208c","Type":"ContainerDied","Data":"9beba3478b7a4089fbe4a41308a21f074819b8ce8d68fa8f6c72d38daaee583a"}
Sep 08 12:03:58.549152 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:58.549052    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx" event={"ID":"4c21df69-1c10-4ee2-aa0d-05764e87208c","Type":"ContainerDied","Data":"a8ff5b0ed509cce318f6f6d7bd0c4210e2158961f2109a86a2f01bce6cab3d3c"}
Sep 08 12:03:58.549152 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:58.549074    2561 scope.go:117] "RemoveContainer" containerID="9beba3478b7a4089fbe4a41308a21f074819b8ce8d68fa8f6c72d38daaee583a"
Sep 08 12:03:58.557315 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:58.557295    2561 scope.go:117] "RemoveContainer" containerID="9beba3478b7a4089fbe4a41308a21f074819b8ce8d68fa8f6c72d38daaee583a"
Sep 08 12:03:58.557586 ip-10-0-139-41 kubenswrapper[2561]: E0908 12:03:58.557570    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9beba3478b7a4089fbe4a41308a21f074819b8ce8d68fa8f6c72d38daaee583a\": container with ID starting with 9beba3478b7a4089fbe4a41308a21f074819b8ce8d68fa8f6c72d38daaee583a not found: ID does not exist" containerID="9beba3478b7a4089fbe4a41308a21f074819b8ce8d68fa8f6c72d38daaee583a"
Sep 08 12:03:58.557650 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:58.557595    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9beba3478b7a4089fbe4a41308a21f074819b8ce8d68fa8f6c72d38daaee583a"} err="failed to get container status \"9beba3478b7a4089fbe4a41308a21f074819b8ce8d68fa8f6c72d38daaee583a\": rpc error: code = NotFound desc = could not find container \"9beba3478b7a4089fbe4a41308a21f074819b8ce8d68fa8f6c72d38daaee583a\": container with ID starting with 9beba3478b7a4089fbe4a41308a21f074819b8ce8d68fa8f6c72d38daaee583a not found: ID does not exist"
Sep 08 12:03:58.571271 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:58.571251    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"]
Sep 08 12:03:58.576004 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:58.575978    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9d73e940-kserve-79d754876c-hq4tx"]
Sep 08 12:03:59.566723 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.566673    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg" podUID="d66f5116-f1c6-4b58-a569-298da915fd7a" containerName="main" containerID="cri-o://f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757" gracePeriod=45
Sep 08 12:03:59.598478 ip-10-0-139-41 kubenswrapper[2561]: E0908 12:03:59.598442    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd66f5116_f1c6_4b58_a569_298da915fd7a.slice/crio-f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757.scope\": RecentStats: unable to find data in memory cache]"
Sep 08 12:03:59.815892 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.815866    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:03:59.911420 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.911330    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-model-cache\") pod \"d66f5116-f1c6-4b58-a569-298da915fd7a\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") "
Sep 08 12:03:59.911420 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.911387    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66f5116-f1c6-4b58-a569-298da915fd7a-tls-certs\") pod \"d66f5116-f1c6-4b58-a569-298da915fd7a\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") "
Sep 08 12:03:59.911420 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.911420    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-tmp-dir\") pod \"d66f5116-f1c6-4b58-a569-298da915fd7a\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") "
Sep 08 12:03:59.911690 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.911456    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bltd9\" (UniqueName: \"kubernetes.io/projected/d66f5116-f1c6-4b58-a569-298da915fd7a-kube-api-access-bltd9\") pod \"d66f5116-f1c6-4b58-a569-298da915fd7a\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") "
Sep 08 12:03:59.911690 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.911520    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-home\") pod \"d66f5116-f1c6-4b58-a569-298da915fd7a\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") "
Sep 08 12:03:59.911690 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.911556    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-dshm\") pod \"d66f5116-f1c6-4b58-a569-298da915fd7a\" (UID: \"d66f5116-f1c6-4b58-a569-298da915fd7a\") "
Sep 08 12:03:59.911853 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.911703    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d66f5116-f1c6-4b58-a569-298da915fd7a" (UID: "d66f5116-f1c6-4b58-a569-298da915fd7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 12:03:59.911853 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.911711    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d66f5116-f1c6-4b58-a569-298da915fd7a" (UID: "d66f5116-f1c6-4b58-a569-298da915fd7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 12:03:59.911853 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.911793    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-tmp-dir\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 12:03:59.911853 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.911812    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-model-cache\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 12:03:59.911853 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.911815    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-home" (OuterVolumeSpecName: "home") pod "d66f5116-f1c6-4b58-a569-298da915fd7a" (UID: "d66f5116-f1c6-4b58-a569-298da915fd7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 12:03:59.913594 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.913563    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-dshm" (OuterVolumeSpecName: "dshm") pod "d66f5116-f1c6-4b58-a569-298da915fd7a" (UID: "d66f5116-f1c6-4b58-a569-298da915fd7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 12:03:59.913594 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.913566    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d66f5116-f1c6-4b58-a569-298da915fd7a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d66f5116-f1c6-4b58-a569-298da915fd7a" (UID: "d66f5116-f1c6-4b58-a569-298da915fd7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 12:03:59.913755 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.913649    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66f5116-f1c6-4b58-a569-298da915fd7a-kube-api-access-bltd9" (OuterVolumeSpecName: "kube-api-access-bltd9") pod "d66f5116-f1c6-4b58-a569-298da915fd7a" (UID: "d66f5116-f1c6-4b58-a569-298da915fd7a"). InnerVolumeSpecName "kube-api-access-bltd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 12:03:59.960119 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:03:59.960086    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c21df69-1c10-4ee2-aa0d-05764e87208c" path="/var/lib/kubelet/pods/4c21df69-1c10-4ee2-aa0d-05764e87208c/volumes"
Sep 08 12:04:00.013051 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:00.013023    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-dshm\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 12:04:00.013051 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:00.013052    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66f5116-f1c6-4b58-a569-298da915fd7a-tls-certs\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 12:04:00.013256 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:00.013068    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bltd9\" (UniqueName: \"kubernetes.io/projected/d66f5116-f1c6-4b58-a569-298da915fd7a-kube-api-access-bltd9\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 12:04:00.013256 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:00.013081    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66f5116-f1c6-4b58-a569-298da915fd7a-home\") on node \"ip-10-0-139-41.ec2.internal\" DevicePath \"\""
Sep 08 12:04:00.559738 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:00.559707    2561 generic.go:356] "Generic (PLEG): container finished" podID="d66f5116-f1c6-4b58-a569-298da915fd7a" containerID="f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757" exitCode=0
Sep 08 12:04:00.559924 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:00.559749    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"
Sep 08 12:04:00.559924 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:00.559777    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg" event={"ID":"d66f5116-f1c6-4b58-a569-298da915fd7a","Type":"ContainerDied","Data":"f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757"}
Sep 08 12:04:00.559924 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:00.559806    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg" event={"ID":"d66f5116-f1c6-4b58-a569-298da915fd7a","Type":"ContainerDied","Data":"155078a00c8189ad36e627537d63e21ad6da0267f022d1693eb4592223c2e03e"}
Sep 08 12:04:00.559924 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:00.559823    2561 scope.go:117] "RemoveContainer" containerID="f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757"
Sep 08 12:04:00.568492 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:00.568271    2561 scope.go:117] "RemoveContainer" containerID="f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757"
Sep 08 12:04:00.568771 ip-10-0-139-41 kubenswrapper[2561]: E0908 12:04:00.568695    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757\": container with ID starting with f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757 not found: ID does not exist" containerID="f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757"
Sep 08 12:04:00.568771 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:00.568723    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757"} err="failed to get container status \"f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757\": rpc error: code = NotFound desc = could not find container \"f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757\": container with ID starting with f12092451d657865053320601df94cf8b243da02eff74cd6d716a6ddacf05757 not found: ID does not exist"
Sep 08 12:04:00.581535 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:00.581513    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"]
Sep 08 12:04:00.589229 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:00.589208    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-9d73e940-kserve-85485bfd47-792hg"]
Sep 08 12:04:01.960266 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:01.960226    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66f5116-f1c6-4b58-a569-298da915fd7a" path="/var/lib/kubelet/pods/d66f5116-f1c6-4b58-a569-298da915fd7a/volumes"
Sep 08 12:04:23.468882 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:23.468755    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 12:04:38.163957 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:38.163920    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-549d4746c7-6jlvm_940ded72-0df4-44fc-8ebd-7bc974ea730e/authorino/0.log"
Sep 08 12:04:44.042366 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:44.042335    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-764984dc4c-ppn5r_7a7207aa-fe29-4074-9d1c-edefc51911e5/manager/0.log"
Sep 08 12:04:44.656958 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:44.656930    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd45fbd48-x2l6s_a746dbca-f2b7-47d3-b876-a9c476945d62/copy-manifests/0.log"
Sep 08 12:04:44.664369 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:44.664345    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd45fbd48-x2l6s_a746dbca-f2b7-47d3-b876-a9c476945d62/manager/0.log"
Sep 08 12:04:45.221436 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:45.221407    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814465-g8hnd_6188b923-04db-43c4-b39f-970012e22779/cleanup/0.log"
Sep 08 12:04:46.215719 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:46.215686    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-549d4746c7-6jlvm_940ded72-0df4-44fc-8ebd-7bc974ea730e/authorino/0.log"
Sep 08 12:04:46.351697 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:46.351666    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 12:04:46.558910 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:46.558834    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-rwc5p_ce05a73a-33fa-4b6f-9d13-95a960e2f0a4/kuadrant-console-plugin/0.log"
Sep 08 12:04:47.624632 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:47.624602    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8d66d765-tr8qv_5f771995-14d3-4418-8046-34f682d39ceb/kube-auth-proxy/0.log"
Sep 08 12:04:47.884995 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:47.884924    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wmp6c_1bf3b339-f292-4de7-94f4-6e80cf89001e/payload-pre-processing/0.log"
Sep 08 12:04:48.460433 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:48.460394    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-qmzsn_d184a11a-9283-4db7-a600-7c3216d6f046/main/0.log"
Sep 08 12:04:48.571082 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:48.571050    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vlm68_79fe915f-6658-4171-b290-0b1741204169/main/0.log"
Sep 08 12:04:49.139318 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:49.139287    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-52k84_fbd4cd91-16a7-4c1a-9444-833b6f13ca72/main/0.log"
Sep 08 12:04:54.522513 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.522454    2561 scope.go:117] "RemoveContainer" containerID="f82d773688be1676c8a372463ba72bb9ad146f141b2778ce53f9792d834cc77c"
Sep 08 12:04:54.638783 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.638746    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ghzlp/must-gather-5cfdk"]
Sep 08 12:04:54.639056 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.639043    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c21df69-1c10-4ee2-aa0d-05764e87208c" containerName="main"
Sep 08 12:04:54.639105 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.639058    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c21df69-1c10-4ee2-aa0d-05764e87208c" containerName="main"
Sep 08 12:04:54.639105 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.639077    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66f5116-f1c6-4b58-a569-298da915fd7a" containerName="main"
Sep 08 12:04:54.639105 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.639083    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66f5116-f1c6-4b58-a569-298da915fd7a" containerName="main"
Sep 08 12:04:54.639201 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.639131    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c21df69-1c10-4ee2-aa0d-05764e87208c" containerName="main"
Sep 08 12:04:54.639201 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.639139    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66f5116-f1c6-4b58-a569-298da915fd7a" containerName="main"
Sep 08 12:04:54.643231 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.643212    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ghzlp/must-gather-5cfdk"
Sep 08 12:04:54.648025 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.648005    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ghzlp\"/\"default-dockercfg-xv459\""
Sep 08 12:04:54.648135 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.648081    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ghzlp\"/\"kube-root-ca.crt\""
Sep 08 12:04:54.648202 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.648182    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ghzlp\"/\"openshift-service-ca.crt\""
Sep 08 12:04:54.655443 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.655414    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ghzlp/must-gather-5cfdk"]
Sep 08 12:04:54.659822 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.659784    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 12:04:54.759674 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.759641    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log" limit=1024
Sep 08 12:04:54.768756 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.768732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd5k2\" (UniqueName: \"kubernetes.io/projected/b04fc47e-94c6-4eac-9fb1-da4ac7f2654d-kube-api-access-sd5k2\") pod \"must-gather-5cfdk\" (UID: \"b04fc47e-94c6-4eac-9fb1-da4ac7f2654d\") " pod="openshift-must-gather-ghzlp/must-gather-5cfdk"
Sep 08 12:04:54.768832 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.768772    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b04fc47e-94c6-4eac-9fb1-da4ac7f2654d-must-gather-output\") pod \"must-gather-5cfdk\" (UID: \"b04fc47e-94c6-4eac-9fb1-da4ac7f2654d\") " pod="openshift-must-gather-ghzlp/must-gather-5cfdk"
Sep 08 12:04:54.870032 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.869950    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd5k2\" (UniqueName: \"kubernetes.io/projected/b04fc47e-94c6-4eac-9fb1-da4ac7f2654d-kube-api-access-sd5k2\") pod \"must-gather-5cfdk\" (UID: \"b04fc47e-94c6-4eac-9fb1-da4ac7f2654d\") " pod="openshift-must-gather-ghzlp/must-gather-5cfdk"
Sep 08 12:04:54.870032 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.869994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b04fc47e-94c6-4eac-9fb1-da4ac7f2654d-must-gather-output\") pod \"must-gather-5cfdk\" (UID: \"b04fc47e-94c6-4eac-9fb1-da4ac7f2654d\") " pod="openshift-must-gather-ghzlp/must-gather-5cfdk"
Sep 08 12:04:54.870377 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.870359    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b04fc47e-94c6-4eac-9fb1-da4ac7f2654d-must-gather-output\") pod \"must-gather-5cfdk\" (UID: \"b04fc47e-94c6-4eac-9fb1-da4ac7f2654d\") " pod="openshift-must-gather-ghzlp/must-gather-5cfdk"
Sep 08 12:04:54.885737 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.885716    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd5k2\" (UniqueName: \"kubernetes.io/projected/b04fc47e-94c6-4eac-9fb1-da4ac7f2654d-kube-api-access-sd5k2\") pod \"must-gather-5cfdk\" (UID: \"b04fc47e-94c6-4eac-9fb1-da4ac7f2654d\") " pod="openshift-must-gather-ghzlp/must-gather-5cfdk"
Sep 08 12:04:54.952698 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:54.952672    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ghzlp/must-gather-5cfdk"
Sep 08 12:04:55.221888 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:55.221793    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ghzlp/must-gather-5cfdk"]
Sep 08 12:04:55.224693 ip-10-0-139-41 kubenswrapper[2561]: W0908 12:04:55.224669    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb04fc47e_94c6_4eac_9fb1_da4ac7f2654d.slice/crio-660875d7bd67d07652451c533862dc460435b750cc1cb0da04ce359271f969fd WatchSource:0}: Error finding container 660875d7bd67d07652451c533862dc460435b750cc1cb0da04ce359271f969fd: Status 404 returned error can't find the container with id 660875d7bd67d07652451c533862dc460435b750cc1cb0da04ce359271f969fd
Sep 08 12:04:55.226956 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:55.226938    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 12:04:55.803610 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:55.803579    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ghzlp/must-gather-5cfdk" event={"ID":"b04fc47e-94c6-4eac-9fb1-da4ac7f2654d","Type":"ContainerStarted","Data":"660875d7bd67d07652451c533862dc460435b750cc1cb0da04ce359271f969fd"}
Sep 08 12:04:56.811077 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:56.811044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ghzlp/must-gather-5cfdk" event={"ID":"b04fc47e-94c6-4eac-9fb1-da4ac7f2654d","Type":"ContainerStarted","Data":"d9b0ed1b89b1855f4c801417442987045ddcafaaea04c9a8e26062acf0d55d2b"}
Sep 08 12:04:56.811571 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:56.811089    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ghzlp/must-gather-5cfdk" event={"ID":"b04fc47e-94c6-4eac-9fb1-da4ac7f2654d","Type":"ContainerStarted","Data":"9fd96f7cb54394a374ebeb477e42e6efccebec08efac13047a89acedf24b4cd8"}
Sep 08 12:04:56.830498 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:56.830427    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ghzlp/must-gather-5cfdk" podStartSLOduration=2.10625984 podStartE2EDuration="2.830410683s" totalImagesPullingTime="724.150843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 12:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 12:04:56.828065087 +0000 UTC m=+2103.394825641" watchObservedRunningTime="2026-09-08 12:04:56.830410683 +0000 UTC m=+2103.397171225"
Sep 08 12:04:57.731512 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:57.731483    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p9x77_0dff2c8b-ae64-424e-9b34-28819a39597d/global-pull-secret-syncer/0.log"
Sep 08 12:04:57.784708 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:57.784670    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4w4xp_50cf80de-8fa6-4300-a09e-4a95c55ef193/konnectivity-agent/0.log"
Sep 08 12:04:57.938938 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:04:57.938910    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-41.ec2.internal_15e2f71567686cf0984f394af0a6f5d0/haproxy/0.log"
Sep 08 12:05:02.183141 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:02.183106    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-549d4746c7-6jlvm_940ded72-0df4-44fc-8ebd-7bc974ea730e/authorino/0.log"
Sep 08 12:05:02.286398 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:02.286370    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-rwc5p_ce05a73a-33fa-4b6f-9d13-95a960e2f0a4/kuadrant-console-plugin/0.log"
Sep 08 12:05:04.631127 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:04.631067    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kwbt2_8614060e-29d7-4d4a-b4a0-a3c52cc2caf5/node-exporter/0.log"
Sep 08 12:05:04.663404 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:04.663342    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kwbt2_8614060e-29d7-4d4a-b4a0-a3c52cc2caf5/kube-rbac-proxy/0.log"
Sep 08 12:05:04.691120 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:04.691087    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kwbt2_8614060e-29d7-4d4a-b4a0-a3c52cc2caf5/init-textfile/0.log"
Sep 08 12:05:06.943428 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:06.943396    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-jdgsc_b6d98443-0a7e-403f-9884-c92a00056280/networking-console-plugin/0.log"
Sep 08 12:05:07.430172 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.430135    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"]
Sep 08 12:05:07.435050 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.435025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.458510 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.458459    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"]
Sep 08 12:05:07.497884 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.497841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccacf00d-182c-48dd-9998-66d9cdc56380-lib-modules\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.498134 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.498116    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqlpz\" (UniqueName: \"kubernetes.io/projected/ccacf00d-182c-48dd-9998-66d9cdc56380-kube-api-access-dqlpz\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.498280 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.498267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccacf00d-182c-48dd-9998-66d9cdc56380-sys\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.498420 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.498407    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ccacf00d-182c-48dd-9998-66d9cdc56380-podres\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.498589 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.498575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ccacf00d-182c-48dd-9998-66d9cdc56380-proc\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.599195 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.599153    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccacf00d-182c-48dd-9998-66d9cdc56380-lib-modules\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.599382 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.599216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqlpz\" (UniqueName: \"kubernetes.io/projected/ccacf00d-182c-48dd-9998-66d9cdc56380-kube-api-access-dqlpz\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.599382 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.599273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccacf00d-182c-48dd-9998-66d9cdc56380-sys\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.599382 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.599302    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ccacf00d-182c-48dd-9998-66d9cdc56380-podres\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.599382 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.599351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ccacf00d-182c-48dd-9998-66d9cdc56380-proc\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.599674 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.599387    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccacf00d-182c-48dd-9998-66d9cdc56380-lib-modules\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.599674 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.599430    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ccacf00d-182c-48dd-9998-66d9cdc56380-proc\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.599674 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.599485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ccacf00d-182c-48dd-9998-66d9cdc56380-podres\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.599674 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.599546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccacf00d-182c-48dd-9998-66d9cdc56380-sys\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.613396 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.613369    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqlpz\" (UniqueName: \"kubernetes.io/projected/ccacf00d-182c-48dd-9998-66d9cdc56380-kube-api-access-dqlpz\") pod \"perf-node-gather-daemonset-7lsqv\" (UID: \"ccacf00d-182c-48dd-9998-66d9cdc56380\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:07.748048 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:07.748014    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:08.049490 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:08.049373    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"]
Sep 08 12:05:08.054647 ip-10-0-139-41 kubenswrapper[2561]: W0908 12:05:08.054616    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podccacf00d_182c_48dd_9998_66d9cdc56380.slice/crio-5166f160c59eaddba4accae79929b3416d2ad58f082294fb4fd34c0159f02488 WatchSource:0}: Error finding container 5166f160c59eaddba4accae79929b3416d2ad58f082294fb4fd34c0159f02488: Status 404 returned error can't find the container with id 5166f160c59eaddba4accae79929b3416d2ad58f082294fb4fd34c0159f02488
Sep 08 12:05:08.892908 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:08.892860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv" event={"ID":"ccacf00d-182c-48dd-9998-66d9cdc56380","Type":"ContainerStarted","Data":"f344fc59b0017641e156628dba2014cba6cee0ff8aac2bc49e2970ba12c04d0f"}
Sep 08 12:05:08.893113 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:08.892936    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv" event={"ID":"ccacf00d-182c-48dd-9998-66d9cdc56380","Type":"ContainerStarted","Data":"5166f160c59eaddba4accae79929b3416d2ad58f082294fb4fd34c0159f02488"}
Sep 08 12:05:08.893753 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:08.893726    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:08.915866 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:08.915808    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv" podStartSLOduration=1.9157886990000002 podStartE2EDuration="1.915788699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 12:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 12:05:08.91455994 +0000 UTC m=+2115.481320481" watchObservedRunningTime="2026-09-08 12:05:08.915788699 +0000 UTC m=+2115.482549241"
Sep 08 12:05:09.220146 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:09.220115    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-tpxvv_110b167e-aed6-4df1-8c00-9aa826338eca/volume-data-source-validator/0.log"
Sep 08 12:05:10.457926 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:10.457885    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-67xmf_1a77e471-aaa5-4f4d-932d-c8466b009a67/dns/0.log"
Sep 08 12:05:10.486922 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:10.486884    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-67xmf_1a77e471-aaa5-4f4d-932d-c8466b009a67/kube-rbac-proxy/0.log"
Sep 08 12:05:10.636809 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:10.636779    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l76h7_00567f83-0336-467f-826b-a57e063c76ac/dns-node-resolver/0.log"
Sep 08 12:05:11.241007 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:11.240981    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5xw68_838a5800-8a91-4b1b-bc39-655796992d82/node-ca/0.log"
Sep 08 12:05:13.052984 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:13.052945    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8d66d765-tr8qv_5f771995-14d3-4418-8046-34f682d39ceb/kube-auth-proxy/0.log"
Sep 08 12:05:13.146024 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:13.145993    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wmp6c_1bf3b339-f292-4de7-94f4-6e80cf89001e/payload-pre-processing/0.log"
Sep 08 12:05:14.107084 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:14.107056    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b5kmv_c229e7b3-1c76-407e-b4cf-6208554a0009/serve-healthcheck-canary/0.log"
Sep 08 12:05:14.764081 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:14.764052    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dclqn_8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f/kube-rbac-proxy/0.log"
Sep 08 12:05:14.796168 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:14.796139    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dclqn_8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f/exporter/0.log"
Sep 08 12:05:14.822569 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:14.822531    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dclqn_8cc57d0a-1e2d-43ce-a7a3-471bb5952c9f/extractor/0.log"
Sep 08 12:05:15.913867 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:15.913834    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-7lsqv"
Sep 08 12:05:17.714121 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:17.714086    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-764984dc4c-ppn5r_7a7207aa-fe29-4074-9d1c-edefc51911e5/manager/0.log"
Sep 08 12:05:18.205110 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:18.205061    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd45fbd48-x2l6s_a746dbca-f2b7-47d3-b876-a9c476945d62/manager/0.log"
Sep 08 12:05:18.274077 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:18.273985    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd45fbd48-x2l6s_a746dbca-f2b7-47d3-b876-a9c476945d62/copy-manifests/0.log"
Sep 08 12:05:18.392940 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:18.392889    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8d66d765-tr8qv_5f771995-14d3-4418-8046-34f682d39ceb/kube-auth-proxy/0.log"
Sep 08 12:05:18.466915 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:18.466883    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wmp6c_1bf3b339-f292-4de7-94f4-6e80cf89001e/payload-pre-processing/0.log"
Sep 08 12:05:19.717386 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:19.717350    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-b949f5b5c-wtpbw_852e9af0-b0b4-40f1-b4ca-a8a39ae5d2b2/manager/0.log"
Sep 08 12:05:19.760145 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:19.760117    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-54s24_e72045e8-41b0-4bdd-a453-9a30cf92794f/openshift-lws-operator/0.log"
Sep 08 12:05:20.676692 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:20.676664    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8d66d765-tr8qv_5f771995-14d3-4418-8046-34f682d39ceb/kube-auth-proxy/0.log"
Sep 08 12:05:20.759579 ip-10-0-139-41 kubenswrapper[2561]: I0908 12:05:20.759510    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wmp6c_1bf3b339-f292-4de7-94f4-6e80cf89001e/payload-pre-processing/0.log"