Sep 18 19:56:36.669297 ip-10-0-132-172 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:56:40.093156 ip-10-0-132-172 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:56:40.093156 ip-10-0-132-172 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:56:40.093156 ip-10-0-132-172 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:56:40.093156 ip-10-0-132-172 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:56:40.093156 ip-10-0-132-172 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:56:40.093900 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.093222    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:56:40.096928 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096911    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096928    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096946    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096949    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096952    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096956    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096960    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096963    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096965    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096968    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096971    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096974    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096976    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096979    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096982    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096985    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:56:40.096979 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096987    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096990    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096993    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096997    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.096999    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097002    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097004    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097007    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097009    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097012    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097014    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097017    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097020    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097022    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097025    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097027    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097030    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097033    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097037    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097040    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:56:40.097364 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097043    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097046    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097048    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097051    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097054    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097056    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097058    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097061    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097064    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097066    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097069    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097071    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097074    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097076    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097079    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097082    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097085    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097088    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097090    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:56:40.097851 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097093    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097096    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097099    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097103    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097107    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097110    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097113    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097115    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097119    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097121    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097124    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097127    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097130    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097133    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097136    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097139    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097142    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097145    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097147    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:56:40.098329 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097150    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097155    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097157    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097160    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097163    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097166    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097170    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097173    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097176    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097179    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097182    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097185    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097188    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097190    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097193    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097196    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097198    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097201    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097204    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097206    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:56:40.098796 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097209    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097211    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097214    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097216    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097219    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097893    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097899    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097902    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097905    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097908    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097911    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097914    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097916    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097919    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097946    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097950    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097952    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097955    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097958    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097960    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:56:40.099303 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097963    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097966    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097969    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097971    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097974    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097977    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097979    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097982    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097984    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097987    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097989    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097992    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097995    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.097999    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098004    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098008    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098011    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098014    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098017    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:56:40.099792 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098020    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098023    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098031    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098034    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098036    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098039    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098042    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098045    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098049    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098051    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098054    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098057    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098059    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098063    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098065    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098068    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098071    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098073    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098076    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098078    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:56:40.100279 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098081    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098083    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098086    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098089    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098091    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098095    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098098    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098101    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098103    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098106    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098109    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098113    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098116    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098119    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098122    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098125    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098127    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098130    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098132    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098135    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:56:40.100765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098138    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098141    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098144    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098146    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098149    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098152    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098154    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098157    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098159    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098162    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098165    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098167    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098170    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098172    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098175    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098178    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098180    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098183    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098186    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:56:40.101300 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098188    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098191    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098194    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098196    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098199    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098201    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098580    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098585    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098589    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098592    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098595    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098597    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098600    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098603    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098606    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098609    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098611    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098614    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098617    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:56:40.101760 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098620    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098622    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098625    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098628    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098630    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098633    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098636    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098638    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098641    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098643    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098646    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098649    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098653    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098657    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098660    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098663    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098666    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098668    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098671    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098673    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:56:40.102230 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098676    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098678    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098681    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098683    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098686    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098688    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098691    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098693    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098696    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098699    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098701    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098704    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098707    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098709    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098712    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098714    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098717    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098719    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098722    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098724    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:56:40.102742 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098727    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098730    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098732    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098735    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098738    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098740    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098743    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098745    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098748    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098751    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098753    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098756    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098758    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098761    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098763    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098766    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098768    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098771    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098773    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:56:40.103261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098776    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098778    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098782    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098785    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098788    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098790    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098793    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098795    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098798    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098801    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098803    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098806    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098809    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098811    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098814    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098817    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098820    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098823    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098827    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098830    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:56:40.103757 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098833    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098837    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098840    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098843    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098846    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098848    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.098851    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.098920    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.098967    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.098973    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.098977    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.098981    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.098984    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.098988    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.098993    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099001    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099004    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099008    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099011    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099015    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099018    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099021    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:56:40.104307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099024    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099028    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099031    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099035    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099038    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099041    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099044    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099048    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099052    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099055    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099058    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099061    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099064    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099067    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099070    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099073    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099076    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099080    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099083    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099086    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099089    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099092    2578 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099099    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099103    2578 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099106    2578 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:56:40.104835 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099109    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099113    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099116    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099120    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099123    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099126    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099129    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099132    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099135    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099137    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099140    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099143    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099146    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099149    2578 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099153    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099156    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099159    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099162    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099165    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099169    2578 flags.go:64] FLAG: --help="false"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099172    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099175    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099178    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:56:40.105432 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099181    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099185    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099188    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099191    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099194    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099197    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099201    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099204    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099208    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099210    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099213    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099217    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099220    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099223    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099226    2578 flags.go:64] FLAG: --lock-file=""
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099229    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099232    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099235    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099240    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099243    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099245    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099248    2578 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099251    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099254    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:56:40.106040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099257    2578 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099260    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099264    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099267    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099271    2578 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099274    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099277    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099280    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099284    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099287    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099290    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099293    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099300    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099303    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099308    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099311    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099314    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099320    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099323    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099326    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099329    2578 flags.go:64] FLAG: --port="10250"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099332    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099335    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ff1cad762cb5b7f5"
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099339    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:56:40.106621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099342    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099345    2578 flags.go:64] FLAG: --register-node="true"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099348    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099351    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099354    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099357    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099360    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099369    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099372    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099375    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099378    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099380    2578 flags.go:64] FLAG: --runonce="false"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099383    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099387    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099390    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099393    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099396    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099399    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099402    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099405    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099408    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099411    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099413    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099418    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099421    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:56:40.107260 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099424    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099427    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099432    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099435    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099439    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099444    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099446    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099449    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099452    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099455    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099458    2578 flags.go:64] FLAG: --v="2"
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099462    2578 flags.go:64] FLAG: --version="false"
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099466    2578 flags.go:64] FLAG: --vmodule=""
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099470    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.099474    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099582    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099586    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099589    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099592    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099594    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099597    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099600    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099602    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:56:40.107859 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099605    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099607    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099610    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099613    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099615    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099618    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099620    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099625    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099627    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099630    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099632    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099635    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099638    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099640    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099644    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099646    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099649    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099651    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099654    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099656    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:56:40.108452 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099659    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099662    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099664    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099669    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099671    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099674    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099676    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099680    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099684    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099688    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099692    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099696    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099699    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099702    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099704    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099707    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099709    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099712    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099714    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:56:40.108951 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099718    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099721    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099724    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099726    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099729    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099731    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099734    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099737    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099740    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099742    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099745    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099747    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099750    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099752    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099755    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099758    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099762    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099764    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099767    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099769    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:56:40.109416 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099772    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099774    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099777    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099780    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099782    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099785    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099788    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099790    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099793    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099796    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099798    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099801    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099805    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099808    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099810    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099813    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099816    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099818    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099821    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099824    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:56:40.109903 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099826    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099829    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099832    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099835    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099837    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099839    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099843    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099847    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099852    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099854    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099857    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.099859    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.100415    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.107210    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.107225    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:56:40.110414 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.107245    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.107254    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107308    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107313    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107316    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107319    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107323    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107327    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107330    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107333    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107336    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107338    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107341    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107344    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107347    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107349    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107352    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107355    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107358    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107360    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:56:40.110820 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107363    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107366    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107369    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107372    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107375    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107378    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107380    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107383    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107386    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107388    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107391    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107395    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107398    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107401    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107404    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107406    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107409    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107411    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107414    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:56:40.111308 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107417    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107419    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107422    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107425    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107427    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107430    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107434    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107438    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107441    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107444    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107447    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107450    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107453    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107455    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107458    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107461    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107464    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107467    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107469    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:56:40.111765 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107472    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107474    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107477    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107479    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107484    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107487    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107491    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107494    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107496    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107499    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107502    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107504    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107507    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107509    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107512    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107515    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107517    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107520    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107523    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:56:40.112261 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107527    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107530    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107533    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107536    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107539    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107542    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107545    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107548    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107551    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107554    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107557    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107560    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107563    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107565    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107568    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107571    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107573    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107576    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107579    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107581    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:56:40.112730 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107587    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107590    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107593    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107596    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.107601    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107712    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107716    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107719    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107722    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107725    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107728    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107731    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107749    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107753    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107756    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:56:40.113218 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107759    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107763    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107766    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107769    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107772    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107774    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107777    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107780    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107783    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107785    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107788    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107790    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107793    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107796    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107798    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107801    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107804    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107807    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107810    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107812    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:56:40.113589 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107815    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107818    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107820    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107823    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107825    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107828    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107831    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107833    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107836    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107838    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107841    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107843    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107846    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107850    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107854    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107857    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107860    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107863    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107866    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107869    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:56:40.114175 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107872    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107875    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107878    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107881    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107884    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107887    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107890    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107892    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107895    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107897    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107900    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107903    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107906    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107908    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107911    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107913    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107917    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107920    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107923    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:56:40.114673 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107926    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107928    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107943    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107946    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107948    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107951    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107953    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107956    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107958    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107961    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107964    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107967    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107969    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107972    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107975    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107978    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107980    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107983    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107993    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:56:40.115155 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107997    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.107999    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.108002    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.108004    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.108007    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.108011    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.108016    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.108019    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.108021    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.108024    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:40.108027    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.108032    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.108134    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.110729    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:56:40.115846 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.111664    2578 server.go:1019] "Starting client certificate rotation"
Sep 18 19:56:40.116380 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.111763    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:56:40.116380 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.112714    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:56:40.116380 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.114015    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:56:40.117254 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.117232    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:56:40.122471 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.122453    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:56:40.124010 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.123995    2578 log.go:25] "Validated CRI v1 image API"
Sep 18 19:56:40.124810 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.124794    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:56:40.127147 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.127129    2578 fs.go:135] Filesystem UUIDs: map[44d085b4-e75b-4f52-91b5-cb633b4fb1a9:/dev/nvme0n1p4 6d92deb8-a7d3-4b64-8b96-a26582424bd0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 18 19:56:40.127212 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.127146    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:56:40.131439 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.131320    2578 manager.go:216] Machine: {Timestamp:2026-09-18 19:56:40.130750009 +0000 UTC m=+1.969213952 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104713 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec224e6f4c7483ea7da13f4945573192 SystemUUID:ec224e6f-4c74-83ea-7da1-3f4945573192 BootID:b591222d-60c8-4f2a-9037-556b2f1a9394 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:ff:bb:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:ff:bb:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:3b:e3:b7:99:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:56:40.131439 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.131433    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:56:40.131574 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.131500    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:56:40.133007 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.132973    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:56:40.133178 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.133013    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-172.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:56:40.134225 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.134214    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:56:40.134268 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.134228    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:56:40.134268 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.134235    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:56:40.134268 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.134245    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:56:40.134351 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.134321    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:56:40.137448 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.137436    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:56:40.137562 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.137552    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:56:40.137598 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.137587    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:56:40.137626 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.137604    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:56:40.137626 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.137618    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:56:40.137701 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.137627    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:56:40.137701 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.137636    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:56:40.138062 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.138047    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:56:40.138152 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.138074    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:56:40.144510 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.144489    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:56:40.146727 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.146714    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:56:40.146770 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.146744    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:56:40.147775 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147758    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:56:40.147874 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147778    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:56:40.147874 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147785    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:56:40.147874 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147791    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:56:40.147874 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147799    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:56:40.147874 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147805    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:56:40.147874 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147818    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:56:40.147874 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147837    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:56:40.147874 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147843    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:56:40.147874 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147848    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:56:40.147874 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147862    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:56:40.147874 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147867    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:56:40.148295 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147894    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:56:40.148295 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.147899    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:56:40.151566 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.151370    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:56:40.151783 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.151760    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:56:40.152200 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.152184    2578 server.go:1268] "Started kubelet"
Sep 18 19:56:40.153462 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.152800    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:56:40.152998 ip-10-0-132-172 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:56:40.154294 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.154117    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:56:40.154294 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.154209    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:56:40.155301 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.155285    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:56:40.155465 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.155450    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:56:40.156636 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.156608    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:56:40.158743 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.158725    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:56:40.159061 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.159046    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:56:40.161008 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.160995    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:56:40.161071 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.161008    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:56:40.161209 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.161157    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-172.ec2.internal\" not found"
Sep 18 19:56:40.161608 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.161588    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:56:40.161608 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.161609    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:56:40.161747 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.161638    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:56:40.161793 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.161765    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:56:40.161793 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.161774    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:56:40.162037 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.162022    2578 factory.go:55] Registering systemd factory
Sep 18 19:56:40.162093 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.162042    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:56:40.162399 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.162383    2578 factory.go:153] Registering CRI-O factory
Sep 18 19:56:40.162531 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.162515    2578 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:56:40.162618 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.162597    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:56:40.162618 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.162617    2578 factory.go:103] Registering Raw factory
Sep 18 19:56:40.162693 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.162635    2578 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:56:40.163319 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.163298    2578 manager.go:349] Starting recovery of all containers
Sep 18 19:56:40.163529 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.163508    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:56:40.163652 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.163632    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:56:40.203598 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.203579    2578 manager.go:354] Recovery completed
Sep 18 19:56:40.207751 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.207736    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:40.208012 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.206767    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-172.ec2.internal.18d68282c8c42a70  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-172.ec2.internal,UID:ip-10-0-132-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-172.ec2.internal,},FirstTimestamp:2026-09-18 19:56:40.152009328 +0000 UTC m=+1.990473274,LastTimestamp:2026-09-18 19:56:40.152009328 +0000 UTC m=+1.990473274,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-172.ec2.internal,}"
Sep 18 19:56:40.209357 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.209331    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:40.209452 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.209363    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:40.209452 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.209379    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:40.210300 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.210286    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:56:40.210300 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.210296    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:56:40.210418 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.210311    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:56:40.210418 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.210290    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:40.211017 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.211004    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:40.211063 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.211031    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:40.211063 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.211040    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:40.211466 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.211452    2578 policy_none.go:49] "None policy: Start"
Sep 18 19:56:40.211505 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.211474    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:56:40.211505 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.211491    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:56:40.212411 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.212399    2578 policy_none.go:47] "Start"
Sep 18 19:56:40.220195 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.220133    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-172.ec2.internal.18d68282cc2f1a47  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-172.ec2.internal,UID:ip-10-0-132-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-172.ec2.internal,},FirstTimestamp:2026-09-18 19:56:40.209349191 +0000 UTC m=+2.047813134,LastTimestamp:2026-09-18 19:56:40.209349191 +0000 UTC m=+2.047813134,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-172.ec2.internal,}"
Sep 18 19:56:40.224033 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.223954    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-172.ec2.internal.18d68282cc2f6c6c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-172.ec2.internal,UID:ip-10-0-132-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-172.ec2.internal,},FirstTimestamp:2026-09-18 19:56:40.20937022 +0000 UTC m=+2.047834168,LastTimestamp:2026-09-18 19:56:40.20937022 +0000 UTC m=+2.047834168,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-172.ec2.internal,}"
Sep 18 19:56:40.228373 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.228314    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-172.ec2.internal.18d68282cc2fa8c8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-172.ec2.internal,UID:ip-10-0-132-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-172.ec2.internal,},FirstTimestamp:2026-09-18 19:56:40.209385672 +0000 UTC m=+2.047849614,LastTimestamp:2026-09-18 19:56:40.209385672 +0000 UTC m=+2.047849614,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-172.ec2.internal,}"
Sep 18 19:56:40.237607 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.237514    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-172.ec2.internal.18d68282cc2f1a47\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-172.ec2.internal.18d68282cc2f1a47  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-172.ec2.internal,UID:ip-10-0-132-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-172.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-172.ec2.internal,},FirstTimestamp:2026-09-18 19:56:40.209349191 +0000 UTC m=+2.047813134,LastTimestamp:2026-09-18 19:56:40.211018289 +0000 UTC m=+2.049482234,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-172.ec2.internal,}"
Sep 18 19:56:40.253264 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.240875    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-172.ec2.internal.18d68282cc2f6c6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-172.ec2.internal.18d68282cc2f6c6c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-172.ec2.internal,UID:ip-10-0-132-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-172.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-172.ec2.internal,},FirstTimestamp:2026-09-18 19:56:40.20937022 +0000 UTC m=+2.047834168,LastTimestamp:2026-09-18 19:56:40.211036568 +0000 UTC m=+2.049500511,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-172.ec2.internal,}"
Sep 18 19:56:40.253264 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.248672    2578 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:56:40.253264 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.248711    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:56:40.253264 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.248726    2578 server.go:87] "Starting device plugin registration server"
Sep 18 19:56:40.253264 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.248713    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-172.ec2.internal.18d68282cc2fa8c8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-172.ec2.internal.18d68282cc2fa8c8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-172.ec2.internal,UID:ip-10-0-132-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-172.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-172.ec2.internal,},FirstTimestamp:2026-09-18 19:56:40.209385672 +0000 UTC m=+2.047849614,LastTimestamp:2026-09-18 19:56:40.211044085 +0000 UTC m=+2.049508027,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-172.ec2.internal,}"
Sep 18 19:56:40.253264 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.248970    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:56:40.253264 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.248990    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:56:40.253264 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.249070    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:56:40.253264 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.249165    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:56:40.253264 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.249174    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:56:40.253264 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.249641    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:56:40.253264 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.249693    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-172.ec2.internal\" not found"
Sep 18 19:56:40.253264 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.252693    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jdsvm"
Sep 18 19:56:40.259496 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.259479    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jdsvm"
Sep 18 19:56:40.310684 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.310645    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:56:40.311858 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.311843    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:56:40.311948 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.311889    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:56:40.311948 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.311908    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:56:40.312064 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.311968    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:56:40.314615 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.314593    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:56:40.349925 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.349879    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:40.351000 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.350983    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:40.351062 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.351015    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:40.351062 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.351025    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:40.351062 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.351048    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.361675 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.361663    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.361737 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.361682    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-172.ec2.internal\": node \"ip-10-0-132-172.ec2.internal\" not found"
Sep 18 19:56:40.413302 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.412686    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-172.ec2.internal"]
Sep 18 19:56:40.413420 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.413350    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:40.414222 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.414208    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:40.414315 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.414240    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:40.414315 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.414256    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:40.416571 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.416557    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:40.416703 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.416688    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.416750 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.416721    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:40.418626 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.418611    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:40.418722 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.418622    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:40.418722 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.418636    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:40.418722 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.418645    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:40.418722 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.418651    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:40.418722 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.418657    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:40.420813 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.420798    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.420890 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.420827    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:56:40.421639 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.421623    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:56:40.421714 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.421648    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:56:40.421714 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.421662    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:56:40.424959 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.424944    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-172.ec2.internal\" not found"
Sep 18 19:56:40.448488 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.448468    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-172.ec2.internal\" not found" node="ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.452725 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.452711    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-172.ec2.internal\" not found" node="ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.464001 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.463977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4dd8f1e48b0203fb7803d21499a5e60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal\" (UID: \"e4dd8f1e48b0203fb7803d21499a5e60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.464060 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.464010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4dd8f1e48b0203fb7803d21499a5e60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal\" (UID: \"e4dd8f1e48b0203fb7803d21499a5e60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.464060 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.464028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/afec21724a7e77cd26c664aec35ac17d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-172.ec2.internal\" (UID: \"afec21724a7e77cd26c664aec35ac17d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.526037 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.526014    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-172.ec2.internal\" not found"
Sep 18 19:56:40.565052 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.565027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4dd8f1e48b0203fb7803d21499a5e60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal\" (UID: \"e4dd8f1e48b0203fb7803d21499a5e60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.565156 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.565060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4dd8f1e48b0203fb7803d21499a5e60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal\" (UID: \"e4dd8f1e48b0203fb7803d21499a5e60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.565156 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.565082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/afec21724a7e77cd26c664aec35ac17d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-172.ec2.internal\" (UID: \"afec21724a7e77cd26c664aec35ac17d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.565156 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.565117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/afec21724a7e77cd26c664aec35ac17d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-172.ec2.internal\" (UID: \"afec21724a7e77cd26c664aec35ac17d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.565156 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.565123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4dd8f1e48b0203fb7803d21499a5e60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal\" (UID: \"e4dd8f1e48b0203fb7803d21499a5e60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.565293 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.565126    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4dd8f1e48b0203fb7803d21499a5e60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal\" (UID: \"e4dd8f1e48b0203fb7803d21499a5e60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.627186 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.627099    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-172.ec2.internal\" not found"
Sep 18 19:56:40.727678 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.727636    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-172.ec2.internal\" not found"
Sep 18 19:56:40.752804 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.752784    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.755961 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.755944    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:40.828501 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.828324    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-172.ec2.internal\" not found"
Sep 18 19:56:40.928958 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:40.928859    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-172.ec2.internal\" not found"
Sep 18 19:56:40.996643 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:40.996615    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:56:41.025745 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.025724    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:56:41.061884 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.061860    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:41.073741 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.073719    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:56:41.074656 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.074644    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal"
Sep 18 19:56:41.091120 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.091105    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:56:41.111275 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.111258    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:56:41.111661 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.111363    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:56:41.111661 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.111363    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:56:41.111661 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.111366    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:56:41.138498 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.138477    2578 apiserver.go:52] "Watching apiserver"
Sep 18 19:56:41.152288 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.152263    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:56:41.152614 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.152597    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-172.ec2.internal","openshift-multus/multus-2fn5b","openshift-multus/multus-additional-cni-plugins-hpkj8","openshift-multus/network-metrics-daemon-6xbfv","openshift-ovn-kubernetes/ovnkube-node-kw5gp","kube-system/global-pull-secret-syncer-mshsc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg","openshift-cluster-node-tuning-operator/tuned-srvvj","openshift-dns/node-resolver-fhfr4","openshift-image-registry/node-ca-7shwk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal","openshift-network-diagnostics/network-check-target-bq6nj","openshift-network-operator/iptables-alerter-llblt","kube-system/konnectivity-agent-8bps7"]
Sep 18 19:56:41.155067 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.155051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.158790 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.158767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-llblt"
Sep 18 19:56:41.159391 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.159373    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:56:41.159391 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.159387    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:56:41.159626 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.159614    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:56:41.159727 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.159714    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gdtkw\""
Sep 18 19:56:41.160818 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.160803    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.161120 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.161101    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:56:41.162944 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.162914    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:41.163025 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.162998    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:56:41.165240 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.165223    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.167371 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.167355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:41.167443 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.167410    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:56:41.168906 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.168891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-etc-selinux\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.168960 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.168914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk2jb\" (UniqueName: \"kubernetes.io/projected/14b2c7a1-79f9-48fe-8273-11b819ece630-kube-api-access-zk2jb\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.168960 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.168943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64efa40b-4a3a-4ca1-808e-051c9c45c6c8-host-slash\") pod \"iptables-alerter-llblt\" (UID: \"64efa40b-4a3a-4ca1-808e-051c9c45c6c8\") " pod="openshift-network-operator/iptables-alerter-llblt"
Sep 18 19:56:41.169031 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.168980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.169031 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.168999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg7jl\" (UniqueName: \"kubernetes.io/projected/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-kube-api-access-rg7jl\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.169031 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-device-dir\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.169031 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqbpg\" (UniqueName: \"kubernetes.io/projected/64efa40b-4a3a-4ca1-808e-051c9c45c6c8-kube-api-access-nqbpg\") pod \"iptables-alerter-llblt\" (UID: \"64efa40b-4a3a-4ca1-808e-051c9c45c6c8\") " pod="openshift-network-operator/iptables-alerter-llblt"
Sep 18 19:56:41.169152 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wms7\" (UniqueName: \"kubernetes.io/projected/0519a79d-05b8-40fe-8d2a-d10b0bf04473-kube-api-access-9wms7\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:41.169152 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-sys-fs\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.169152 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-cnibin\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.169152 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.169152 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.169152 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.169361 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-socket-dir\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.169361 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64efa40b-4a3a-4ca1-808e-051c9c45c6c8-iptables-alerter-script\") pod \"iptables-alerter-llblt\" (UID: \"64efa40b-4a3a-4ca1-808e-051c9c45c6c8\") " pod="openshift-network-operator/iptables-alerter-llblt"
Sep 18 19:56:41.169361 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-system-cni-dir\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.169361 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.169361 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:41.169361 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-registration-dir\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.169544 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.169380    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fhfr4"
Sep 18 19:56:41.172329 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.172314    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8bps7"
Sep 18 19:56:41.174539 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.174524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.176779 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.176765    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7shwk"
Sep 18 19:56:41.177605 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.177476    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:56:41.177605 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.177489    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:56:41.177605 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.177504    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:56:41.177605 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.177506    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:56:41.177605 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.177566    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:56:41.177893 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.177791    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:56:41.177893 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.177873    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:56:41.178003 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.177950    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:56:41.178410 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178088    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:56:41.178410 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178059    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:56:41.178410 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178108    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:56:41.178410 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178273    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sb9rw\""
Sep 18 19:56:41.178410 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178314    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:56:41.178410 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178322    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:56:41.178410 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178277    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:56:41.178720 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178475    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6zxtn\""
Sep 18 19:56:41.178720 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178543    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:56:41.178818 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178780    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:56:41.178818 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178795    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jb267\""
Sep 18 19:56:41.178818 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178815    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w29br\""
Sep 18 19:56:41.179000 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178864    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:56:41.179000 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178914    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lfcjv\""
Sep 18 19:56:41.179000 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.178964    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:56:41.179138 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.179007    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:56:41.179138 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.179006    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xk7s5\""
Sep 18 19:56:41.179560 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.179541    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.179837 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.179812    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:56:41.180146 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.180131    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tdcqh\""
Sep 18 19:56:41.180830 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.180803    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:56:41.180921 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.180831    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:56:41.181922 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.181906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:41.181992 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.181974    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:56:41.184572 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.184525    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:56:41.184680 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.184572    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:56:41.185042 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.185024    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:56:41.185119 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.185054    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ml6zn\""
Sep 18 19:56:41.187868 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.187852    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:56:41.238821 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.238622    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fz956"
Sep 18 19:56:41.247591 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.247572    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fz956"
Sep 18 19:56:41.262582 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.262559    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:56:41.264268 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.264203    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:51:40 +0000 UTC" deadline="2028-05-23 01:55:51.7521148 +0000 UTC"
Sep 18 19:56:41.264268 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.264266    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14693h59m10.487852779s"
Sep 18 19:56:41.269990 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.269970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-socket-dir\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.270093 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5470bc7-569a-490e-8ac3-e63b5738b859-hosts-file\") pod \"node-resolver-fhfr4\" (UID: \"a5470bc7-569a-490e-8ac3-e63b5738b859\") " pod="openshift-dns/node-resolver-fhfr4"
Sep 18 19:56:41.270093 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9cd771fc-4b1c-49ca-a5b7-b31be69f17fd-konnectivity-ca\") pod \"konnectivity-agent-8bps7\" (UID: \"9cd771fc-4b1c-49ca-a5b7-b31be69f17fd\") " pod="kube-system/konnectivity-agent-8bps7"
Sep 18 19:56:41.270093 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-systemd\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.270212 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-slash\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.270212 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-cni-bin\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.270212 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-socket-dir\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.270212 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-registration-dir\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.270212 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-etc-selinux\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.270212 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5470bc7-569a-490e-8ac3-e63b5738b859-tmp-dir\") pod \"node-resolver-fhfr4\" (UID: \"a5470bc7-569a-490e-8ac3-e63b5738b859\") " pod="openshift-dns/node-resolver-fhfr4"
Sep 18 19:56:41.270212 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270191    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-registration-dir\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.270512 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-sysctl-d\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.270512 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270262    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-etc-selinux\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.270512 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-run\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.270512 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg7bh\" (UniqueName: \"kubernetes.io/projected/71246950-0f11-4069-a03d-06dd6d3aeb16-kube-api-access-kg7bh\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.270512 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sczdl\" (UniqueName: \"kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl\") pod \"network-check-target-bq6nj\" (UID: \"ac0567c9-870f-417a-a157-98253c4850b3\") " pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:41.270512 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-system-cni-dir\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.270512 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-device-dir\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.270512 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-sysconfig\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.270512 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-host\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.270512 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-run-systemd\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.270512 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-device-dir\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.270512 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-run-netns\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.270512 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-var-lib-kubelet\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-run-multus-certs\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-cnibin\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9cd771fc-4b1c-49ca-a5b7-b31be69f17fd-agent-certs\") pod \"konnectivity-agent-8bps7\" (UID: \"9cd771fc-4b1c-49ca-a5b7-b31be69f17fd\") " pod="kube-system/konnectivity-agent-8bps7"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270624    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-var-lib-kubelet\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-cnibin\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270681    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-systemd-units\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-run-k8s-cni-cncf-io\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64efa40b-4a3a-4ca1-808e-051c9c45c6c8-iptables-alerter-script\") pod \"iptables-alerter-llblt\" (UID: \"64efa40b-4a3a-4ca1-808e-051c9c45c6c8\") " pod="openshift-network-operator/iptables-alerter-llblt"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-system-cni-dir\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-sysctl-conf\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71246950-0f11-4069-a03d-06dd6d3aeb16-tmp\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-system-cni-dir\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-var-lib-openvswitch\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64efa40b-4a3a-4ca1-808e-051c9c45c6c8-host-slash\") pod \"iptables-alerter-llblt\" (UID: \"64efa40b-4a3a-4ca1-808e-051c9c45c6c8\") " pod="openshift-network-operator/iptables-alerter-llblt"
Sep 18 19:56:41.271050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270882    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64efa40b-4a3a-4ca1-808e-051c9c45c6c8-host-slash\") pod \"iptables-alerter-llblt\" (UID: \"64efa40b-4a3a-4ca1-808e-051c9c45c6c8\") " pod="openshift-network-operator/iptables-alerter-llblt"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1685cb44-5bb4-493c-b414-cb4e2ec0ab2f-host\") pod \"node-ca-7shwk\" (UID: \"1685cb44-5bb4-493c-b414-cb4e2ec0ab2f\") " pod="openshift-image-registry/node-ca-7shwk"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-run-netns\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-run-ovn-kubernetes\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.270990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84b8e4eb-2d7c-4c5c-902a-91db799dc858-ovn-node-metrics-cert\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxqk\" (UniqueName: \"kubernetes.io/projected/84b8e4eb-2d7c-4c5c-902a-91db799dc858-kube-api-access-bfxqk\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271043    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-multus-socket-dir-parent\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-sys-fs\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-kubelet\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-multus-conf-dir\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14b2c7a1-79f9-48fe-8273-11b819ece630-sys-fs\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg7jl\" (UniqueName: \"kubernetes.io/projected/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-kube-api-access-rg7jl\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-run-ovn\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-multus-cni-dir\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e63fbdf1-33c7-45c7-b40a-3537e53da4be-cni-binary-copy\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271258    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-sys\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.271829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84b8e4eb-2d7c-4c5c-902a-91db799dc858-ovnkube-script-lib\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea005810-7434-414a-8805-b1a2032e8108-dbus\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-var-lib-cni-bin\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1685cb44-5bb4-493c-b414-cb4e2ec0ab2f-serviceca\") pod \"node-ca-7shwk\" (UID: \"1685cb44-5bb4-493c-b414-cb4e2ec0ab2f\") " pod="openshift-image-registry/node-ca-7shwk"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgzpb\" (UniqueName: \"kubernetes.io/projected/a5470bc7-569a-490e-8ac3-e63b5738b859-kube-api-access-cgzpb\") pod \"node-resolver-fhfr4\" (UID: \"a5470bc7-569a-490e-8ac3-e63b5738b859\") " pod="openshift-dns/node-resolver-fhfr4"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84b8e4eb-2d7c-4c5c-902a-91db799dc858-env-overrides\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/64efa40b-4a3a-4ca1-808e-051c9c45c6c8-iptables-alerter-script\") pod \"iptables-alerter-llblt\" (UID: \"64efa40b-4a3a-4ca1-808e-051c9c45c6c8\") " pod="openshift-network-operator/iptables-alerter-llblt"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-etc-kubernetes\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-modprobe-d\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-node-log\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcv2q\" (UniqueName: \"kubernetes.io/projected/1685cb44-5bb4-493c-b414-cb4e2ec0ab2f-kube-api-access-xcv2q\") pod \"node-ca-7shwk\" (UID: \"1685cb44-5bb4-493c-b414-cb4e2ec0ab2f\") " pod="openshift-image-registry/node-ca-7shwk"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271628    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-etc-openvswitch\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.272611 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-var-lib-cni-multus\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e63fbdf1-33c7-45c7-b40a-3537e53da4be-multus-daemon-config\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk2jb\" (UniqueName: \"kubernetes.io/projected/14b2c7a1-79f9-48fe-8273-11b819ece630-kube-api-access-zk2jb\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-lib-modules\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-log-socket\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.271852    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84b8e4eb-2d7c-4c5c-902a-91db799dc858-ovnkube-config\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-hostroot\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.271912    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs podName:0519a79d-05b8-40fe-8d2a-d10b0bf04473 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:41.771891443 +0000 UTC m=+3.610355394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs") pod "network-metrics-daemon-6xbfv" (UID: "0519a79d-05b8-40fe-8d2a-d10b0bf04473") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzz4r\" (UniqueName: \"kubernetes.io/projected/e63fbdf1-33c7-45c7-b40a-3537e53da4be-kube-api-access-lzz4r\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.271978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqbpg\" (UniqueName: \"kubernetes.io/projected/64efa40b-4a3a-4ca1-808e-051c9c45c6c8-kube-api-access-nqbpg\") pod \"iptables-alerter-llblt\" (UID: \"64efa40b-4a3a-4ca1-808e-051c9c45c6c8\") " pod="openshift-network-operator/iptables-alerter-llblt"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.272003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-cni-netd\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.272022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea005810-7434-414a-8805-b1a2032e8108-kubelet-config\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.272044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wms7\" (UniqueName: \"kubernetes.io/projected/0519a79d-05b8-40fe-8d2a-d10b0bf04473-kube-api-access-9wms7\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:41.273083 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.272060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.273576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.272060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.273576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.272075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-kubernetes\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.273576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.272089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-tuned\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.273576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.272116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-run-openvswitch\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.273576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.272152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.273576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.272176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-cnibin\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.273576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.272376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.279293 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.279275    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:56:41.282349 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.282329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk2jb\" (UniqueName: \"kubernetes.io/projected/14b2c7a1-79f9-48fe-8273-11b819ece630-kube-api-access-zk2jb\") pod \"aws-ebs-csi-driver-node-tgsbg\" (UID: \"14b2c7a1-79f9-48fe-8273-11b819ece630\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.282460 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.282357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg7jl\" (UniqueName: \"kubernetes.io/projected/2652ec9f-d178-4d1d-b37d-b0644b52b7b3-kube-api-access-rg7jl\") pod \"multus-additional-cni-plugins-hpkj8\" (UID: \"2652ec9f-d178-4d1d-b37d-b0644b52b7b3\") " pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.286217 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.286196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wms7\" (UniqueName: \"kubernetes.io/projected/0519a79d-05b8-40fe-8d2a-d10b0bf04473-kube-api-access-9wms7\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:41.294202 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.294184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqbpg\" (UniqueName: \"kubernetes.io/projected/64efa40b-4a3a-4ca1-808e-051c9c45c6c8-kube-api-access-nqbpg\") pod \"iptables-alerter-llblt\" (UID: \"64efa40b-4a3a-4ca1-808e-051c9c45c6c8\") " pod="openshift-network-operator/iptables-alerter-llblt"
Sep 18 19:56:41.372433 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-sysctl-conf\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.372433 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71246950-0f11-4069-a03d-06dd6d3aeb16-tmp\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.372599 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-var-lib-openvswitch\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.372599 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1685cb44-5bb4-493c-b414-cb4e2ec0ab2f-host\") pod \"node-ca-7shwk\" (UID: \"1685cb44-5bb4-493c-b414-cb4e2ec0ab2f\") " pod="openshift-image-registry/node-ca-7shwk"
Sep 18 19:56:41.372599 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-run-netns\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.372599 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-run-ovn-kubernetes\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.372599 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-var-lib-openvswitch\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.372599 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84b8e4eb-2d7c-4c5c-902a-91db799dc858-ovn-node-metrics-cert\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.372599 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-run-netns\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.372599 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372561    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-sysctl-conf\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.372599 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxqk\" (UniqueName: \"kubernetes.io/projected/84b8e4eb-2d7c-4c5c-902a-91db799dc858-kube-api-access-bfxqk\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-multus-socket-dir-parent\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1685cb44-5bb4-493c-b414-cb4e2ec0ab2f-host\") pod \"node-ca-7shwk\" (UID: \"1685cb44-5bb4-493c-b414-cb4e2ec0ab2f\") " pod="openshift-image-registry/node-ca-7shwk"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-kubelet\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372553    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-run-ovn-kubernetes\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-multus-conf-dir\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-kubelet\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-run-ovn\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-run-ovn\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372729    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-multus-socket-dir-parent\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-multus-cni-dir\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-multus-conf-dir\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e63fbdf1-33c7-45c7-b40a-3537e53da4be-cni-binary-copy\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-sys\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-multus-cni-dir\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84b8e4eb-2d7c-4c5c-902a-91db799dc858-ovnkube-script-lib\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea005810-7434-414a-8805-b1a2032e8108-dbus\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-sys\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.373035 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-var-lib-cni-bin\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1685cb44-5bb4-493c-b414-cb4e2ec0ab2f-serviceca\") pod \"node-ca-7shwk\" (UID: \"1685cb44-5bb4-493c-b414-cb4e2ec0ab2f\") " pod="openshift-image-registry/node-ca-7shwk"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgzpb\" (UniqueName: \"kubernetes.io/projected/a5470bc7-569a-490e-8ac3-e63b5738b859-kube-api-access-cgzpb\") pod \"node-resolver-fhfr4\" (UID: \"a5470bc7-569a-490e-8ac3-e63b5738b859\") " pod="openshift-dns/node-resolver-fhfr4"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84b8e4eb-2d7c-4c5c-902a-91db799dc858-env-overrides\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-etc-kubernetes\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373040    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-modprobe-d\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-node-log\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcv2q\" (UniqueName: \"kubernetes.io/projected/1685cb44-5bb4-493c-b414-cb4e2ec0ab2f-kube-api-access-xcv2q\") pod \"node-ca-7shwk\" (UID: \"1685cb44-5bb4-493c-b414-cb4e2ec0ab2f\") " pod="openshift-image-registry/node-ca-7shwk"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-etc-openvswitch\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-var-lib-cni-multus\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e63fbdf1-33c7-45c7-b40a-3537e53da4be-multus-daemon-config\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-lib-modules\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-var-lib-cni-bin\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-log-socket\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84b8e4eb-2d7c-4c5c-902a-91db799dc858-ovnkube-config\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.372927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea005810-7434-414a-8805-b1a2032e8108-dbus\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-hostroot\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.374028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzz4r\" (UniqueName: \"kubernetes.io/projected/e63fbdf1-33c7-45c7-b40a-3537e53da4be-kube-api-access-lzz4r\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-cni-netd\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea005810-7434-414a-8805-b1a2032e8108-kubelet-config\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1685cb44-5bb4-493c-b414-cb4e2ec0ab2f-serviceca\") pod \"node-ca-7shwk\" (UID: \"1685cb44-5bb4-493c-b414-cb4e2ec0ab2f\") " pod="openshift-image-registry/node-ca-7shwk"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-kubernetes\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-tuned\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-run-openvswitch\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84b8e4eb-2d7c-4c5c-902a-91db799dc858-ovnkube-script-lib\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-etc-kubernetes\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.373487    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373490    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea005810-7434-414a-8805-b1a2032e8108-kubelet-config\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373499    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-cnibin\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5470bc7-569a-490e-8ac3-e63b5738b859-hosts-file\") pod \"node-resolver-fhfr4\" (UID: \"a5470bc7-569a-490e-8ac3-e63b5738b859\") " pod="openshift-dns/node-resolver-fhfr4"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-kubernetes\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9cd771fc-4b1c-49ca-a5b7-b31be69f17fd-konnectivity-ca\") pod \"konnectivity-agent-8bps7\" (UID: \"9cd771fc-4b1c-49ca-a5b7-b31be69f17fd\") " pod="kube-system/konnectivity-agent-8bps7"
Sep 18 19:56:41.374885 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.373585    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret podName:ea005810-7434-414a-8805-b1a2032e8108 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:41.873567262 +0000 UTC m=+3.712031207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret") pod "global-pull-secret-syncer-mshsc" (UID: "ea005810-7434-414a-8805-b1a2032e8108") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e63fbdf1-33c7-45c7-b40a-3537e53da4be-cni-binary-copy\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-etc-openvswitch\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84b8e4eb-2d7c-4c5c-902a-91db799dc858-env-overrides\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-systemd\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-slash\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-node-log\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-cni-bin\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-cni-bin\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5470bc7-569a-490e-8ac3-e63b5738b859-tmp-dir\") pod \"node-resolver-fhfr4\" (UID: \"a5470bc7-569a-490e-8ac3-e63b5738b859\") " pod="openshift-dns/node-resolver-fhfr4"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-sysctl-d\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-var-lib-cni-multus\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-run\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373871    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-hostroot\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg7bh\" (UniqueName: \"kubernetes.io/projected/71246950-0f11-4069-a03d-06dd6d3aeb16-kube-api-access-kg7bh\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-lib-modules\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczdl\" (UniqueName: \"kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl\") pod \"network-check-target-bq6nj\" (UID: \"ac0567c9-870f-417a-a157-98253c4850b3\") " pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373967    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-log-socket\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.375559 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-system-cni-dir\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-run-openvswitch\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.373998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-sysctl-d\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-sysconfig\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9cd771fc-4b1c-49ca-a5b7-b31be69f17fd-konnectivity-ca\") pod \"konnectivity-agent-8bps7\" (UID: \"9cd771fc-4b1c-49ca-a5b7-b31be69f17fd\") " pod="kube-system/konnectivity-agent-8bps7"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-system-cni-dir\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374075    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-systemd\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-host\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-run-systemd\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-run-netns\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-var-lib-kubelet\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-run-multus-certs\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9cd771fc-4b1c-49ca-a5b7-b31be69f17fd-agent-certs\") pod \"konnectivity-agent-8bps7\" (UID: \"9cd771fc-4b1c-49ca-a5b7-b31be69f17fd\") " pod="kube-system/konnectivity-agent-8bps7"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5470bc7-569a-490e-8ac3-e63b5738b859-tmp-dir\") pod \"node-resolver-fhfr4\" (UID: \"a5470bc7-569a-490e-8ac3-e63b5738b859\") " pod="openshift-dns/node-resolver-fhfr4"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-cni-netd\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5470bc7-569a-490e-8ac3-e63b5738b859-hosts-file\") pod \"node-resolver-fhfr4\" (UID: \"a5470bc7-569a-490e-8ac3-e63b5738b859\") " pod="openshift-dns/node-resolver-fhfr4"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84b8e4eb-2d7c-4c5c-902a-91db799dc858-ovnkube-config\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-modprobe-d\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-host\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-run-systemd\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374663    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-run\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e63fbdf1-33c7-45c7-b40a-3537e53da4be-multus-daemon-config\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-sysconfig\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-host-slash\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-var-lib-kubelet\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-systemd-units\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-var-lib-kubelet\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374777    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-cnibin\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374798    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-run-multus-certs\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84b8e4eb-2d7c-4c5c-902a-91db799dc858-systemd-units\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71246950-0f11-4069-a03d-06dd6d3aeb16-var-lib-kubelet\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-run-k8s-cni-cncf-io\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374882    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-run-netns\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.374957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e63fbdf1-33c7-45c7-b40a-3537e53da4be-host-run-k8s-cni-cncf-io\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.375137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71246950-0f11-4069-a03d-06dd6d3aeb16-tmp\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.375233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84b8e4eb-2d7c-4c5c-902a-91db799dc858-ovn-node-metrics-cert\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.376576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.375593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/71246950-0f11-4069-a03d-06dd6d3aeb16-etc-tuned\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.377107 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.376299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9cd771fc-4b1c-49ca-a5b7-b31be69f17fd-agent-certs\") pod \"konnectivity-agent-8bps7\" (UID: \"9cd771fc-4b1c-49ca-a5b7-b31be69f17fd\") " pod="kube-system/konnectivity-agent-8bps7"
Sep 18 19:56:41.384350 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.384332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxqk\" (UniqueName: \"kubernetes.io/projected/84b8e4eb-2d7c-4c5c-902a-91db799dc858-kube-api-access-bfxqk\") pod \"ovnkube-node-kw5gp\" (UID: \"84b8e4eb-2d7c-4c5c-902a-91db799dc858\") " pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.385466 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.385450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgzpb\" (UniqueName: \"kubernetes.io/projected/a5470bc7-569a-490e-8ac3-e63b5738b859-kube-api-access-cgzpb\") pod \"node-resolver-fhfr4\" (UID: \"a5470bc7-569a-490e-8ac3-e63b5738b859\") " pod="openshift-dns/node-resolver-fhfr4"
Sep 18 19:56:41.392159 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.392141    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg7bh\" (UniqueName: \"kubernetes.io/projected/71246950-0f11-4069-a03d-06dd6d3aeb16-kube-api-access-kg7bh\") pod \"tuned-srvvj\" (UID: \"71246950-0f11-4069-a03d-06dd6d3aeb16\") " pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.392995 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.392978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzz4r\" (UniqueName: \"kubernetes.io/projected/e63fbdf1-33c7-45c7-b40a-3537e53da4be-kube-api-access-lzz4r\") pod \"multus-2fn5b\" (UID: \"e63fbdf1-33c7-45c7-b40a-3537e53da4be\") " pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.393557 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.393542    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:56:41.393616 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.393565    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:56:41.393616 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.393575    2578 projected.go:196] Error preparing data for projected volume kube-api-access-sczdl for pod openshift-network-diagnostics/network-check-target-bq6nj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:41.393682 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.393629    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl podName:ac0567c9-870f-417a-a157-98253c4850b3 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:41.893614913 +0000 UTC m=+3.732078842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sczdl" (UniqueName: "kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl") pod "network-check-target-bq6nj" (UID: "ac0567c9-870f-417a-a157-98253c4850b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:41.394694 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.394674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcv2q\" (UniqueName: \"kubernetes.io/projected/1685cb44-5bb4-493c-b414-cb4e2ec0ab2f-kube-api-access-xcv2q\") pod \"node-ca-7shwk\" (UID: \"1685cb44-5bb4-493c-b414-cb4e2ec0ab2f\") " pod="openshift-image-registry/node-ca-7shwk"
Sep 18 19:56:41.399321 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:41.399299    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4dd8f1e48b0203fb7803d21499a5e60.slice/crio-14a594dc067a21a2a3d4ec2c57ba484c00f778907d0e3311ac59cc4a318781f2 WatchSource:0}: Error finding container 14a594dc067a21a2a3d4ec2c57ba484c00f778907d0e3311ac59cc4a318781f2: Status 404 returned error can't find the container with id 14a594dc067a21a2a3d4ec2c57ba484c00f778907d0e3311ac59cc4a318781f2
Sep 18 19:56:41.399610 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:41.399594    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafec21724a7e77cd26c664aec35ac17d.slice/crio-30257ecb94604ab42e2cea3c08e73694ef24edac04e7c4ca0d873a0e51622f70 WatchSource:0}: Error finding container 30257ecb94604ab42e2cea3c08e73694ef24edac04e7c4ca0d873a0e51622f70: Status 404 returned error can't find the container with id 30257ecb94604ab42e2cea3c08e73694ef24edac04e7c4ca0d873a0e51622f70
Sep 18 19:56:41.405989 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.405972    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:56:41.488852 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.488827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg"
Sep 18 19:56:41.498366 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:41.498340    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b2c7a1_79f9_48fe_8273_11b819ece630.slice/crio-58e7e661b5f39b85ac8b7b6bee72b9971145ffd5fbd66eeba8fe7e38d6e4b5d4 WatchSource:0}: Error finding container 58e7e661b5f39b85ac8b7b6bee72b9971145ffd5fbd66eeba8fe7e38d6e4b5d4: Status 404 returned error can't find the container with id 58e7e661b5f39b85ac8b7b6bee72b9971145ffd5fbd66eeba8fe7e38d6e4b5d4
Sep 18 19:56:41.501371 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.501355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-llblt"
Sep 18 19:56:41.509356 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:41.509335    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64efa40b_4a3a_4ca1_808e_051c9c45c6c8.slice/crio-d11121a99edd935ff5144e0100bc38e42ac7b7574611bd0ec7838f46939562cf WatchSource:0}: Error finding container d11121a99edd935ff5144e0100bc38e42ac7b7574611bd0ec7838f46939562cf: Status 404 returned error can't find the container with id d11121a99edd935ff5144e0100bc38e42ac7b7574611bd0ec7838f46939562cf
Sep 18 19:56:41.514082 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.514064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hpkj8"
Sep 18 19:56:41.523803 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:41.523784    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2652ec9f_d178_4d1d_b37d_b0644b52b7b3.slice/crio-2bc8d133193534039d684fb02b9645cbe373559409710b0e7ac61cc6c8e847c9 WatchSource:0}: Error finding container 2bc8d133193534039d684fb02b9645cbe373559409710b0e7ac61cc6c8e847c9: Status 404 returned error can't find the container with id 2bc8d133193534039d684fb02b9645cbe373559409710b0e7ac61cc6c8e847c9
Sep 18 19:56:41.528079 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.528059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:56:41.536286 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:41.536267    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b8e4eb_2d7c_4c5c_902a_91db799dc858.slice/crio-a67499a527da16c6aa64c85adbd58da80d6aed03f695d02b00f0bddccb02ca94 WatchSource:0}: Error finding container a67499a527da16c6aa64c85adbd58da80d6aed03f695d02b00f0bddccb02ca94: Status 404 returned error can't find the container with id a67499a527da16c6aa64c85adbd58da80d6aed03f695d02b00f0bddccb02ca94
Sep 18 19:56:41.549862 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.549844    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fhfr4"
Sep 18 19:56:41.557923 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:41.557901    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5470bc7_569a_490e_8ac3_e63b5738b859.slice/crio-ca348c974658fdb1fb4346c72e81da742472e80f1a131b0e97ec49d248b2cd28 WatchSource:0}: Error finding container ca348c974658fdb1fb4346c72e81da742472e80f1a131b0e97ec49d248b2cd28: Status 404 returned error can't find the container with id ca348c974658fdb1fb4346c72e81da742472e80f1a131b0e97ec49d248b2cd28
Sep 18 19:56:41.563708 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.563693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8bps7"
Sep 18 19:56:41.571029 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:41.571009    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd771fc_4b1c_49ca_a5b7_b31be69f17fd.slice/crio-fc8dbd06a67997588126eac9f6234214efb0c14dcf69b53801dc743c9218dbab WatchSource:0}: Error finding container fc8dbd06a67997588126eac9f6234214efb0c14dcf69b53801dc743c9218dbab: Status 404 returned error can't find the container with id fc8dbd06a67997588126eac9f6234214efb0c14dcf69b53801dc743c9218dbab
Sep 18 19:56:41.582027 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.582001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2fn5b"
Sep 18 19:56:41.591235 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:41.591208    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63fbdf1_33c7_45c7_b40a_3537e53da4be.slice/crio-7aabba05df95207620f735eea38ef99735e9ca1db2f7ddc05b3073f087b66b95 WatchSource:0}: Error finding container 7aabba05df95207620f735eea38ef99735e9ca1db2f7ddc05b3073f087b66b95: Status 404 returned error can't find the container with id 7aabba05df95207620f735eea38ef99735e9ca1db2f7ddc05b3073f087b66b95
Sep 18 19:56:41.600903 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.600886    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7shwk"
Sep 18 19:56:41.605534 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.605519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-srvvj"
Sep 18 19:56:41.609664 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:41.609644    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1685cb44_5bb4_493c_b414_cb4e2ec0ab2f.slice/crio-418ee1a31e7392ae4bfbc88d117986541a112b5f2f750b9ec815504dbb76e83e WatchSource:0}: Error finding container 418ee1a31e7392ae4bfbc88d117986541a112b5f2f750b9ec815504dbb76e83e: Status 404 returned error can't find the container with id 418ee1a31e7392ae4bfbc88d117986541a112b5f2f750b9ec815504dbb76e83e
Sep 18 19:56:41.613911 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:56:41.613892    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71246950_0f11_4069_a03d_06dd6d3aeb16.slice/crio-7b38928f6578b387c79591c8d22cb617ceb41bcdcbbc52410dad93d2c3a7f744 WatchSource:0}: Error finding container 7b38928f6578b387c79591c8d22cb617ceb41bcdcbbc52410dad93d2c3a7f744: Status 404 returned error can't find the container with id 7b38928f6578b387c79591c8d22cb617ceb41bcdcbbc52410dad93d2c3a7f744
Sep 18 19:56:41.778537 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.778468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:41.778674 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.778585    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:41.778674 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.778668    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs podName:0519a79d-05b8-40fe-8d2a-d10b0bf04473 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:42.778650711 +0000 UTC m=+4.617114643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs") pod "network-metrics-daemon-6xbfv" (UID: "0519a79d-05b8-40fe-8d2a-d10b0bf04473") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:41.878888 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.878849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:41.879069 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.879014    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:41.879132 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.879091    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret podName:ea005810-7434-414a-8805-b1a2032e8108 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:42.879069267 +0000 UTC m=+4.717533212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret") pod "global-pull-secret-syncer-mshsc" (UID: "ea005810-7434-414a-8805-b1a2032e8108") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:41.979919 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:41.979884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczdl\" (UniqueName: \"kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl\") pod \"network-check-target-bq6nj\" (UID: \"ac0567c9-870f-417a-a157-98253c4850b3\") " pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:41.980133 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.980116    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:56:41.980202 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.980141    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:56:41.980202 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.980153    2578 projected.go:196] Error preparing data for projected volume kube-api-access-sczdl for pod openshift-network-diagnostics/network-check-target-bq6nj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:41.980302 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:41.980216    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl podName:ac0567c9-870f-417a-a157-98253c4850b3 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:42.980190052 +0000 UTC m=+4.818653987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sczdl" (UniqueName: "kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl") pod "network-check-target-bq6nj" (UID: "ac0567c9-870f-417a-a157-98253c4850b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:42.250796 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.250726    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:51:41 +0000 UTC" deadline="2028-06-07 22:34:02.86656981 +0000 UTC"
Sep 18 19:56:42.250796 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.250785    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15074h37m20.61578853s"
Sep 18 19:56:42.314953 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.314905    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:42.315109 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:42.315040    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:56:42.322482 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.322455    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:56:42.338818 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.338765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2fn5b" event={"ID":"e63fbdf1-33c7-45c7-b40a-3537e53da4be","Type":"ContainerStarted","Data":"7aabba05df95207620f735eea38ef99735e9ca1db2f7ddc05b3073f087b66b95"}
Sep 18 19:56:42.363282 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.363205    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpkj8" event={"ID":"2652ec9f-d178-4d1d-b37d-b0644b52b7b3","Type":"ContainerStarted","Data":"2bc8d133193534039d684fb02b9645cbe373559409710b0e7ac61cc6c8e847c9"}
Sep 18 19:56:42.382956 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.380279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-llblt" event={"ID":"64efa40b-4a3a-4ca1-808e-051c9c45c6c8","Type":"ContainerStarted","Data":"d11121a99edd935ff5144e0100bc38e42ac7b7574611bd0ec7838f46939562cf"}
Sep 18 19:56:42.398311 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.398276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg" event={"ID":"14b2c7a1-79f9-48fe-8273-11b819ece630","Type":"ContainerStarted","Data":"58e7e661b5f39b85ac8b7b6bee72b9971145ffd5fbd66eeba8fe7e38d6e4b5d4"}
Sep 18 19:56:42.426176 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.426139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7shwk" event={"ID":"1685cb44-5bb4-493c-b414-cb4e2ec0ab2f","Type":"ContainerStarted","Data":"418ee1a31e7392ae4bfbc88d117986541a112b5f2f750b9ec815504dbb76e83e"}
Sep 18 19:56:42.443318 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.443284    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8bps7" event={"ID":"9cd771fc-4b1c-49ca-a5b7-b31be69f17fd","Type":"ContainerStarted","Data":"fc8dbd06a67997588126eac9f6234214efb0c14dcf69b53801dc743c9218dbab"}
Sep 18 19:56:42.462772 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.462696    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fhfr4" event={"ID":"a5470bc7-569a-490e-8ac3-e63b5738b859","Type":"ContainerStarted","Data":"ca348c974658fdb1fb4346c72e81da742472e80f1a131b0e97ec49d248b2cd28"}
Sep 18 19:56:42.474704 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.474580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp" event={"ID":"84b8e4eb-2d7c-4c5c-902a-91db799dc858","Type":"ContainerStarted","Data":"a67499a527da16c6aa64c85adbd58da80d6aed03f695d02b00f0bddccb02ca94"}
Sep 18 19:56:42.488399 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.488165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal" event={"ID":"e4dd8f1e48b0203fb7803d21499a5e60","Type":"ContainerStarted","Data":"14a594dc067a21a2a3d4ec2c57ba484c00f778907d0e3311ac59cc4a318781f2"}
Sep 18 19:56:42.499250 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.499180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-172.ec2.internal" event={"ID":"afec21724a7e77cd26c664aec35ac17d","Type":"ContainerStarted","Data":"30257ecb94604ab42e2cea3c08e73694ef24edac04e7c4ca0d873a0e51622f70"}
Sep 18 19:56:42.512212 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.512142    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-srvvj" event={"ID":"71246950-0f11-4069-a03d-06dd6d3aeb16","Type":"ContainerStarted","Data":"7b38928f6578b387c79591c8d22cb617ceb41bcdcbbc52410dad93d2c3a7f744"}
Sep 18 19:56:42.786844 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.786744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:42.787055 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:42.786926    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:42.787055 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:42.787009    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs podName:0519a79d-05b8-40fe-8d2a-d10b0bf04473 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:44.786987442 +0000 UTC m=+6.625451391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs") pod "network-metrics-daemon-6xbfv" (UID: "0519a79d-05b8-40fe-8d2a-d10b0bf04473") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:42.887993 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.887945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:42.888177 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:42.888147    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:42.888239 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:42.888209    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret podName:ea005810-7434-414a-8805-b1a2032e8108 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:44.888190705 +0000 UTC m=+6.726654641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret") pod "global-pull-secret-syncer-mshsc" (UID: "ea005810-7434-414a-8805-b1a2032e8108") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:42.988480 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:42.988440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczdl\" (UniqueName: \"kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl\") pod \"network-check-target-bq6nj\" (UID: \"ac0567c9-870f-417a-a157-98253c4850b3\") " pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:42.988644 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:42.988593    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:56:42.988644 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:42.988614    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:56:42.988644 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:42.988627    2578 projected.go:196] Error preparing data for projected volume kube-api-access-sczdl for pod openshift-network-diagnostics/network-check-target-bq6nj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:42.988827 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:42.988686    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl podName:ac0567c9-870f-417a-a157-98253c4850b3 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:44.988668297 +0000 UTC m=+6.827132241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sczdl" (UniqueName: "kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl") pod "network-check-target-bq6nj" (UID: "ac0567c9-870f-417a-a157-98253c4850b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:43.251313 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:43.251268    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:51:41 +0000 UTC" deadline="2028-03-19 07:02:51.020690237 +0000 UTC"
Sep 18 19:56:43.251313 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:43.251311    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13139h6m7.76938342s"
Sep 18 19:56:43.312534 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:43.312503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:43.312704 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:43.312636    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:56:43.313112 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:43.313082    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:43.313210 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:43.313188    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:56:43.564549 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:43.564474    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:56:44.127445 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:44.127415    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:56:44.315303 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:44.315268    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:44.315759 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:44.315395    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:56:44.804567 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:44.804526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:44.804767 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:44.804745    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:44.804834 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:44.804821    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs podName:0519a79d-05b8-40fe-8d2a-d10b0bf04473 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:48.804801289 +0000 UTC m=+10.643265246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs") pod "network-metrics-daemon-6xbfv" (UID: "0519a79d-05b8-40fe-8d2a-d10b0bf04473") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:44.906402 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:44.906027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:44.906402 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:44.906227    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:44.906402 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:44.906288    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret podName:ea005810-7434-414a-8805-b1a2032e8108 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:48.906269512 +0000 UTC m=+10.744733457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret") pod "global-pull-secret-syncer-mshsc" (UID: "ea005810-7434-414a-8805-b1a2032e8108") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:45.006901 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:45.006796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczdl\" (UniqueName: \"kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl\") pod \"network-check-target-bq6nj\" (UID: \"ac0567c9-870f-417a-a157-98253c4850b3\") " pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:45.007105 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:45.007014    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:56:45.007105 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:45.007042    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:56:45.007105 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:45.007056    2578 projected.go:196] Error preparing data for projected volume kube-api-access-sczdl for pod openshift-network-diagnostics/network-check-target-bq6nj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:45.007268 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:45.007161    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl podName:ac0567c9-870f-417a-a157-98253c4850b3 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:49.007141384 +0000 UTC m=+10.845605318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sczdl" (UniqueName: "kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl") pod "network-check-target-bq6nj" (UID: "ac0567c9-870f-417a-a157-98253c4850b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:45.313001 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:45.312966    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:45.313187 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:45.312972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:45.313187 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:45.313131    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:56:45.313303 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:45.313186    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:56:46.312771 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:46.312646    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:46.313264 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:46.312777    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:56:47.313100 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:47.313065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:47.313100 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:47.313093    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:47.313649 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:47.313187    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:56:47.313649 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:47.313379    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:56:48.313203 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:48.313108    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:48.313637 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:48.313242    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:56:48.837656 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:48.837623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:48.837819 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:48.837794    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:48.837874 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:48.837864    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs podName:0519a79d-05b8-40fe-8d2a-d10b0bf04473 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:56.837844166 +0000 UTC m=+18.676308102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs") pod "network-metrics-daemon-6xbfv" (UID: "0519a79d-05b8-40fe-8d2a-d10b0bf04473") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:48.938328 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:48.938263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:48.938545 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:48.938437    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:48.938545 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:48.938512    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret podName:ea005810-7434-414a-8805-b1a2032e8108 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:56.938496693 +0000 UTC m=+18.776960647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret") pod "global-pull-secret-syncer-mshsc" (UID: "ea005810-7434-414a-8805-b1a2032e8108") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:49.039197 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:49.039160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczdl\" (UniqueName: \"kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl\") pod \"network-check-target-bq6nj\" (UID: \"ac0567c9-870f-417a-a157-98253c4850b3\") " pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:49.039378 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:49.039324    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:56:49.039378 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:49.039346    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:56:49.039378 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:49.039357    2578 projected.go:196] Error preparing data for projected volume kube-api-access-sczdl for pod openshift-network-diagnostics/network-check-target-bq6nj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:49.039525 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:49.039417    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl podName:ac0567c9-870f-417a-a157-98253c4850b3 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:57.039397958 +0000 UTC m=+18.877861889 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sczdl" (UniqueName: "kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl") pod "network-check-target-bq6nj" (UID: "ac0567c9-870f-417a-a157-98253c4850b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:49.312600 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:49.312513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:49.312794 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:49.312646    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:56:49.313029 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:49.312955    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:49.313154 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:49.313076    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:56:50.315989 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:50.315953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:50.316426 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:50.316093    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:56:51.312365 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:51.312328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:51.312557 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:51.312374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:51.312557 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:51.312474    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:56:51.312675 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:51.312614    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:56:52.312962 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:52.312914    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:52.313394 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:52.313037    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:56:53.312402 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:53.312368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:53.312688 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:53.312496    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:56:53.312688 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:53.312556    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:53.312796 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:53.312685    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:56:54.315655 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:54.315613    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:54.316126 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:54.315723    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:56:55.312417 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:55.312383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:55.312589 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:55.312385    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:55.312589 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:55.312515    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:56:55.312713 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:55.312598    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:56:56.312622 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:56.312588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:56.313051 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:56.312724    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:56:56.896324 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:56.896288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:56.896521 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:56.896430    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:56.896521 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:56.896493    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs podName:0519a79d-05b8-40fe-8d2a-d10b0bf04473 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:12.896473255 +0000 UTC m=+34.734937184 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs") pod "network-metrics-daemon-6xbfv" (UID: "0519a79d-05b8-40fe-8d2a-d10b0bf04473") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:56:56.997189 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:56.997153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:56.997352 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:56.997293    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:56.997401 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:56.997365    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret podName:ea005810-7434-414a-8805-b1a2032e8108 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:12.997343502 +0000 UTC m=+34.835807446 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret") pod "global-pull-secret-syncer-mshsc" (UID: "ea005810-7434-414a-8805-b1a2032e8108") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:56:57.098273 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:57.098242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczdl\" (UniqueName: \"kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl\") pod \"network-check-target-bq6nj\" (UID: \"ac0567c9-870f-417a-a157-98253c4850b3\") " pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:57.098441 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:57.098403    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:56:57.098441 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:57.098423    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:56:57.098441 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:57.098435    2578 projected.go:196] Error preparing data for projected volume kube-api-access-sczdl for pod openshift-network-diagnostics/network-check-target-bq6nj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:57.098560 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:57.098497    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl podName:ac0567c9-870f-417a-a157-98253c4850b3 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:13.098480564 +0000 UTC m=+34.936944495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sczdl" (UniqueName: "kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl") pod "network-check-target-bq6nj" (UID: "ac0567c9-870f-417a-a157-98253c4850b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:56:57.312974 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:57.312927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:57.313496 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:57.312927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:57.313496 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:57.313081    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:56:57.313496 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:57.313157    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:56:58.312947 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:58.312901    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:56:58.313118 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:58.313048    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:56:59.312700 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:59.312665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:56:59.312884 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:59.312783    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:56:59.312884 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:56:59.312837    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:56:59.313003 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:56:59.312953    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:57:00.313408 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:00.313375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:00.313837 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:00.313483    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:57:01.313120 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.312792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:57:01.313244 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.312824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:57:01.313244 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:01.313202    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:57:01.313367 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:01.313337    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:57:01.558412 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.558368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7shwk" event={"ID":"1685cb44-5bb4-493c-b414-cb4e2ec0ab2f","Type":"ContainerStarted","Data":"28c0b0438d41f6b51ae7e351fbf9158fc20c36012c500db127daef2f5314beb6"}
Sep 18 19:57:01.559916 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.559885    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8bps7" event={"ID":"9cd771fc-4b1c-49ca-a5b7-b31be69f17fd","Type":"ContainerStarted","Data":"af428585ea6991d3e0903a65a88a0a23fc2d4afb7f07fef0f23053e3909c2c92"}
Sep 18 19:57:01.561431 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.561407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fhfr4" event={"ID":"a5470bc7-569a-490e-8ac3-e63b5738b859","Type":"ContainerStarted","Data":"89c3b5500a8ff1856255e7bfa9b0613c0289c96489112f52a9a43100164006b3"}
Sep 18 19:57:01.564558 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.564525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp" event={"ID":"84b8e4eb-2d7c-4c5c-902a-91db799dc858","Type":"ContainerStarted","Data":"90ca70a7765f06ac9a795df2f0f05d7e6b08fae3aa3f6f87d833018c71e879c2"}
Sep 18 19:57:01.564649 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.564598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp" event={"ID":"84b8e4eb-2d7c-4c5c-902a-91db799dc858","Type":"ContainerStarted","Data":"71b5a449e4a44103ebb854807eedfa58c8678b4a6dd2d4bb0508683605573bb3"}
Sep 18 19:57:01.564694 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.564651    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp" event={"ID":"84b8e4eb-2d7c-4c5c-902a-91db799dc858","Type":"ContainerStarted","Data":"cdc54f62753c9e499d4f04440efc2910e9d725482a4c91ca25244abf683a01fc"}
Sep 18 19:57:01.564733 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.564698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp" event={"ID":"84b8e4eb-2d7c-4c5c-902a-91db799dc858","Type":"ContainerStarted","Data":"11878e28a1d9dbe70696305d11cd01ce4be2187e7a92a6582f274d04bbe72dcf"}
Sep 18 19:57:01.564770 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.564743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp" event={"ID":"84b8e4eb-2d7c-4c5c-902a-91db799dc858","Type":"ContainerStarted","Data":"ae3f5855c679f7d1862706d6912943003e772e5b1071c86cea7206236b859a78"}
Sep 18 19:57:01.564810 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.564780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp" event={"ID":"84b8e4eb-2d7c-4c5c-902a-91db799dc858","Type":"ContainerStarted","Data":"0320103762de79558ab017a9834dab224ac1e16fcd3bb4e8c6a6c94e3588994f"}
Sep 18 19:57:01.566157 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.566129    2578 generic.go:356] "Generic (PLEG): container finished" podID="e4dd8f1e48b0203fb7803d21499a5e60" containerID="5b29e21a0f346ae7ba7c73a2875cbc23d0097fc0e708017e0768e5b12607fd25" exitCode=0
Sep 18 19:57:01.566245 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.566174    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal" event={"ID":"e4dd8f1e48b0203fb7803d21499a5e60","Type":"ContainerDied","Data":"5b29e21a0f346ae7ba7c73a2875cbc23d0097fc0e708017e0768e5b12607fd25"}
Sep 18 19:57:01.567587 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.567566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-172.ec2.internal" event={"ID":"afec21724a7e77cd26c664aec35ac17d","Type":"ContainerStarted","Data":"262aaeae17f245c1fb2c3d88c77981ac7691b01a5859e2adfccee627b789821b"}
Sep 18 19:57:01.568926 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.568907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-srvvj" event={"ID":"71246950-0f11-4069-a03d-06dd6d3aeb16","Type":"ContainerStarted","Data":"636ad55cbf80ea00ecfad8f6d07f9c6db89e0959404667825e0d54032275f374"}
Sep 18 19:57:01.570470 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.570447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2fn5b" event={"ID":"e63fbdf1-33c7-45c7-b40a-3537e53da4be","Type":"ContainerStarted","Data":"cb9bb9bc3657a9d256681e115fa2ca318109b130697d32f47d86e68550a743f7"}
Sep 18 19:57:01.571918 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.571900    2578 generic.go:356] "Generic (PLEG): container finished" podID="2652ec9f-d178-4d1d-b37d-b0644b52b7b3" containerID="d309bbaa001f081d3aa159f0fbf4d1786cccd1c15e04f26abc938e8c4f24e465" exitCode=0
Sep 18 19:57:01.572026 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.571962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpkj8" event={"ID":"2652ec9f-d178-4d1d-b37d-b0644b52b7b3","Type":"ContainerDied","Data":"d309bbaa001f081d3aa159f0fbf4d1786cccd1c15e04f26abc938e8c4f24e465"}
Sep 18 19:57:01.573371 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.573350    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg" event={"ID":"14b2c7a1-79f9-48fe-8273-11b819ece630","Type":"ContainerStarted","Data":"14918d5c23cf20d9ac082c835bdbc7ee5337f1af4bfcaa79f53984af951b539e"}
Sep 18 19:57:01.577072 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.577034    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7shwk" podStartSLOduration=2.672235186 podStartE2EDuration="21.577009326s" totalImagesPullingTime="18.90477414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:01.576538731 +0000 UTC m=+23.415002683" watchObservedRunningTime="2026-09-18 19:57:01.577009326 +0000 UTC m=+23.415473278"
Sep 18 19:57:01.597706 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.597671    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-srvvj" podStartSLOduration=2.6665768 podStartE2EDuration="21.597659806s" totalImagesPullingTime="18.931083006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:01.597240526 +0000 UTC m=+23.435704478" watchObservedRunningTime="2026-09-18 19:57:01.597659806 +0000 UTC m=+23.436123790"
Sep 18 19:57:01.619766 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.619715    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-172.ec2.internal" podStartSLOduration=20.619704364 podStartE2EDuration="20.619704364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:01.619333197 +0000 UTC m=+23.457797151" watchObservedRunningTime="2026-09-18 19:57:01.619704364 +0000 UTC m=+23.458168316"
Sep 18 19:57:01.679841 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.679784    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fhfr4" podStartSLOduration=2.722794667 podStartE2EDuration="21.679764499s" totalImagesPullingTime="18.956969832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:01.679503389 +0000 UTC m=+23.517967342" watchObservedRunningTime="2026-09-18 19:57:01.679764499 +0000 UTC m=+23.518228454"
Sep 18 19:57:01.698355 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.698301    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8bps7" podStartSLOduration=2.725905344 podStartE2EDuration="21.698280592s" totalImagesPullingTime="18.972375248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:01.698212555 +0000 UTC m=+23.536676507" watchObservedRunningTime="2026-09-18 19:57:01.698280592 +0000 UTC m=+23.536744545"
Sep 18 19:57:01.718410 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:01.718359    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2fn5b" podStartSLOduration=2.75996755 podStartE2EDuration="21.718339763s" totalImagesPullingTime="18.958372213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:01.718110821 +0000 UTC m=+23.556574774" watchObservedRunningTime="2026-09-18 19:57:01.718339763 +0000 UTC m=+23.556803718"
Sep 18 19:57:02.040734 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:02.040581    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:57:02.261704 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:02.261573    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:57:02.040730536Z","UUID":"dd6a7ab7-f111-4c01-badc-3cf7c2076774","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:57:02.263207 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:02.263188    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:57:02.263307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:02.263224    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:57:02.313010 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:02.312987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:02.313155 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:02.313087    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:57:02.338593 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:02.338565    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8bps7"
Sep 18 19:57:02.339204 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:02.339185    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8bps7"
Sep 18 19:57:02.578240 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:02.578152    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg" event={"ID":"14b2c7a1-79f9-48fe-8273-11b819ece630","Type":"ContainerStarted","Data":"7321e20548918057f1d61611bb48b5659c72b950198d496ec6bc088d426316b2"}
Sep 18 19:57:02.580263 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:02.580233    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal" event={"ID":"e4dd8f1e48b0203fb7803d21499a5e60","Type":"ContainerStarted","Data":"5a2ff9fd8fa909142dd7b450637037cabbeaa573d532b8cc2d808be6cf7892d8"}
Sep 18 19:57:02.585732 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:02.585687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-llblt" event={"ID":"64efa40b-4a3a-4ca1-808e-051c9c45c6c8","Type":"ContainerStarted","Data":"e10e88a00ade8ebcd0b74a036102accef361ed2cfa001de125c7c90f1684bfd6"}
Sep 18 19:57:02.616966 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:02.616896    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-172.ec2.internal" podStartSLOduration=21.616880306 podStartE2EDuration="21.616880306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:02.616278977 +0000 UTC m=+24.454742932" watchObservedRunningTime="2026-09-18 19:57:02.616880306 +0000 UTC m=+24.455344258"
Sep 18 19:57:02.634712 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:02.634662    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-llblt" podStartSLOduration=3.628561771 podStartE2EDuration="22.634643298s" totalImagesPullingTime="19.006081527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:02.634251305 +0000 UTC m=+24.472715387" watchObservedRunningTime="2026-09-18 19:57:02.634643298 +0000 UTC m=+24.473107252"
Sep 18 19:57:03.313115 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:03.313090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:57:03.313242 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:03.313117    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:57:03.313292 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:03.313231    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:57:03.313359 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:03.313341    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:57:03.590672 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:03.590535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg" event={"ID":"14b2c7a1-79f9-48fe-8273-11b819ece630","Type":"ContainerStarted","Data":"f0a1ed7af79ed5e318aac086b3b45cdaceb868012a83b755810759eece1e3a3a"}
Sep 18 19:57:03.594819 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:03.594788    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:57:03.594819 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:03.594787    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp" event={"ID":"84b8e4eb-2d7c-4c5c-902a-91db799dc858","Type":"ContainerStarted","Data":"fefe78dc56c4fd3dde35295216e872ad52b816b0afa6cc2dbe2fdd55d32b1217"}
Sep 18 19:57:03.612876 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:03.612823    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgsbg" podStartSLOduration=2.728136665 podStartE2EDuration="23.612804892s" totalImagesPullingTime="20.884668227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:03.611488335 +0000 UTC m=+25.449952290" watchObservedRunningTime="2026-09-18 19:57:03.612804892 +0000 UTC m=+25.451268848"
Sep 18 19:57:04.315786 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:04.315757    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:04.315964 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:04.315853    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:57:05.313066 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:05.313029    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:57:05.313575 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:05.313077    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:57:05.313575 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:05.313168    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:57:05.313575 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:05.313307    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:57:06.312928 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:06.312768    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:06.313069 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:06.313020    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:57:06.605448 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:06.605408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp" event={"ID":"84b8e4eb-2d7c-4c5c-902a-91db799dc858","Type":"ContainerStarted","Data":"98fed100c9c452b572749fc1e7a5f61ecf205452c2602b2ba73d24f9cff63013"}
Sep 18 19:57:06.606110 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:06.605642    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:57:06.607423 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:06.607396    2578 generic.go:356] "Generic (PLEG): container finished" podID="2652ec9f-d178-4d1d-b37d-b0644b52b7b3" containerID="f7a52eecaa6c0b943bed76e8d8631892a38586e948bbb9818172738fedaa6f20" exitCode=0
Sep 18 19:57:06.607509 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:06.607438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpkj8" event={"ID":"2652ec9f-d178-4d1d-b37d-b0644b52b7b3","Type":"ContainerDied","Data":"f7a52eecaa6c0b943bed76e8d8631892a38586e948bbb9818172738fedaa6f20"}
Sep 18 19:57:06.620966 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:06.620945    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:57:06.638218 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:06.638183    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp" podStartSLOduration=7.512497248 podStartE2EDuration="26.638170819s" totalImagesPullingTime="19.125673571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:06.637353813 +0000 UTC m=+28.475817764" watchObservedRunningTime="2026-09-18 19:57:06.638170819 +0000 UTC m=+28.476634770"
Sep 18 19:57:07.312179 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:07.312151    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:57:07.312353 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:07.312287    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:57:07.312353 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:07.312340    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:57:07.312523 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:07.312495    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:57:07.612162 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:07.612090    2578 generic.go:356] "Generic (PLEG): container finished" podID="2652ec9f-d178-4d1d-b37d-b0644b52b7b3" containerID="e95eaf0630dc0de26f8b8cacaa2c55fdbe92b6714edc6d37be8c0f0eee38ea48" exitCode=0
Sep 18 19:57:07.612641 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:07.612156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpkj8" event={"ID":"2652ec9f-d178-4d1d-b37d-b0644b52b7b3","Type":"ContainerDied","Data":"e95eaf0630dc0de26f8b8cacaa2c55fdbe92b6714edc6d37be8c0f0eee38ea48"}
Sep 18 19:57:07.612701 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:07.612662    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:57:07.612744 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:07.612732    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:57:07.627280 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:07.627256    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:57:07.844724 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:07.844696    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bq6nj"]
Sep 18 19:57:07.844885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:07.844798    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:07.844970 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:07.844884    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:57:07.847191 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:07.847163    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mshsc"]
Sep 18 19:57:07.847308 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:07.847258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:57:07.847366 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:07.847350    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:57:07.847789 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:07.847772    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xbfv"]
Sep 18 19:57:07.847860 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:07.847848    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:57:07.847959 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:07.847923    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:57:08.617268 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:08.617201    2578 generic.go:356] "Generic (PLEG): container finished" podID="2652ec9f-d178-4d1d-b37d-b0644b52b7b3" containerID="d8cd79387973e12013847636686fb5a4f3ae1b758df26ec8e5c85cf595a185a3" exitCode=0
Sep 18 19:57:08.617631 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:08.617284    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpkj8" event={"ID":"2652ec9f-d178-4d1d-b37d-b0644b52b7b3","Type":"ContainerDied","Data":"d8cd79387973e12013847636686fb5a4f3ae1b758df26ec8e5c85cf595a185a3"}
Sep 18 19:57:09.312922 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:09.312869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:09.312922 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:09.312916    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:57:09.313187 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:09.312917    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:57:09.313187 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:09.313013    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:57:09.313187 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:09.313093    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:57:09.313187 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:09.313162    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:57:10.715207 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:10.715016    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8bps7"
Sep 18 19:57:10.715654 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:10.715326    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 19:57:10.715709 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:10.715676    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8bps7"
Sep 18 19:57:11.312897 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:11.312862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:11.313073 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:11.312908    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:57:11.313073 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:11.312976    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:57:11.313073 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:11.312994    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:57:11.313073 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:11.313053    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:57:11.313217 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:11.313116    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:57:12.907365 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:12.907322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:57:12.907868 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:12.907450    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:57:12.907868 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:12.907516    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs podName:0519a79d-05b8-40fe-8d2a-d10b0bf04473 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:44.907494853 +0000 UTC m=+66.745958784 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs") pod "network-metrics-daemon-6xbfv" (UID: "0519a79d-05b8-40fe-8d2a-d10b0bf04473") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:57:13.008368 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.008332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:57:13.008533 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.008487    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:57:13.008576 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.008543    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret podName:ea005810-7434-414a-8805-b1a2032e8108 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:45.008528629 +0000 UTC m=+66.846992578 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret") pod "global-pull-secret-syncer-mshsc" (UID: "ea005810-7434-414a-8805-b1a2032e8108") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:57:13.109307 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.109271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczdl\" (UniqueName: \"kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl\") pod \"network-check-target-bq6nj\" (UID: \"ac0567c9-870f-417a-a157-98253c4850b3\") " pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:13.109490 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.109402    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:57:13.109490 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.109418    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:57:13.109490 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.109429    2578 projected.go:196] Error preparing data for projected volume kube-api-access-sczdl for pod openshift-network-diagnostics/network-check-target-bq6nj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:57:13.109490 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.109481    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl podName:ac0567c9-870f-417a-a157-98253c4850b3 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:45.109468117 +0000 UTC m=+66.947932047 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sczdl" (UniqueName: "kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl") pod "network-check-target-bq6nj" (UID: "ac0567c9-870f-417a-a157-98253c4850b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:57:13.312986 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.312955    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:13.313169 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.313065    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:57:13.313169 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.313074    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq6nj" podUID="ac0567c9-870f-417a-a157-98253c4850b3"
Sep 18 19:57:13.313287 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.313167    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mshsc" podUID="ea005810-7434-414a-8805-b1a2032e8108"
Sep 18 19:57:13.313287 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.313074    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:57:13.313287 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.313254    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbfv" podUID="0519a79d-05b8-40fe-8d2a-d10b0bf04473"
Sep 18 19:57:13.509798 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.509775    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-172.ec2.internal" event="NodeReady"
Sep 18 19:57:13.510015 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.509894    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:57:13.577863 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.577835    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d4d89d599-dqphx"]
Sep 18 19:57:13.599608 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.599585    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.604497 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.604472    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d4d89d599-dqphx"]
Sep 18 19:57:13.606247 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.606223    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 19:57:13.606396 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.606377    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 19:57:13.607351 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.606959    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 19:57:13.610646 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.610630    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-466kg\""
Sep 18 19:57:13.637458 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.637438    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9pkl9"]
Sep 18 19:57:13.643366 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.643340    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 19:57:13.652050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.652028    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9pkl9"
Sep 18 19:57:13.654683 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.654665    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9h4s5"]
Sep 18 19:57:13.659131 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.659112    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:57:13.659227 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.659211    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xx2wd\""
Sep 18 19:57:13.659664 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.659648    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:57:13.659746 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.659665    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:57:13.674406 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.674390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:13.679281 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.679261    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:57:13.679447 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.679403    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9pkl9"]
Sep 18 19:57:13.679716 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.679697    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r2wjd\""
Sep 18 19:57:13.680064 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.680043    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9h4s5"]
Sep 18 19:57:13.684096 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.684077    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:57:13.713170 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.713150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-bound-sa-token\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.713280 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.713180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.713280 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.713212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afae54d8-af97-45fd-92b2-155b4db81db6-trusted-ca\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.713361 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.713291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23-cert\") pod \"ingress-canary-9pkl9\" (UID: \"6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23\") " pod="openshift-ingress-canary/ingress-canary-9pkl9"
Sep 18 19:57:13.713361 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.713313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afae54d8-af97-45fd-92b2-155b4db81db6-registry-certificates\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.713361 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.713334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afae54d8-af97-45fd-92b2-155b4db81db6-image-registry-private-configuration\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.713493 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.713365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afae54d8-af97-45fd-92b2-155b4db81db6-ca-trust-extracted\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.713493 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.713451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afae54d8-af97-45fd-92b2-155b4db81db6-installation-pull-secrets\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.713590 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.713490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmsd9\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-kube-api-access-cmsd9\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.713590 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.713526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vg97\" (UniqueName: \"kubernetes.io/projected/6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23-kube-api-access-7vg97\") pod \"ingress-canary-9pkl9\" (UID: \"6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23\") " pod="openshift-ingress-canary/ingress-canary-9pkl9"
Sep 18 19:57:13.814565 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.814497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afae54d8-af97-45fd-92b2-155b4db81db6-installation-pull-secrets\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.814565 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.814528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmsd9\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-kube-api-access-cmsd9\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.814565 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.814548    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:13.814565 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.814565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vg97\" (UniqueName: \"kubernetes.io/projected/6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23-kube-api-access-7vg97\") pod \"ingress-canary-9pkl9\" (UID: \"6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23\") " pod="openshift-ingress-canary/ingress-canary-9pkl9"
Sep 18 19:57:13.814854 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.814594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-bound-sa-token\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.814854 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.814733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.814854 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.814828    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:57:13.814854 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.814845    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4d89d599-dqphx: secret "image-registry-tls" not found
Sep 18 19:57:13.814854 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.814846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afae54d8-af97-45fd-92b2-155b4db81db6-trusted-ca\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.815165 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.814889    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls podName:afae54d8-af97-45fd-92b2-155b4db81db6 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:14.31487339 +0000 UTC m=+36.153337325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls") pod "image-registry-d4d89d599-dqphx" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6") : secret "image-registry-tls" not found
Sep 18 19:57:13.815165 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.814925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23-cert\") pod \"ingress-canary-9pkl9\" (UID: \"6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23\") " pod="openshift-ingress-canary/ingress-canary-9pkl9"
Sep 18 19:57:13.815165 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.814978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afae54d8-af97-45fd-92b2-155b4db81db6-registry-certificates\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.815165 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.815013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9cb99c-d1fd-46f8-ae30-981161eaf053-config-volume\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:13.815165 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.815143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a9cb99c-d1fd-46f8-ae30-981161eaf053-tmp-dir\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:13.815403 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.815194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afae54d8-af97-45fd-92b2-155b4db81db6-image-registry-private-configuration\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.815403 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.815255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m74l\" (UniqueName: \"kubernetes.io/projected/6a9cb99c-d1fd-46f8-ae30-981161eaf053-kube-api-access-2m74l\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:13.815403 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.815303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afae54d8-af97-45fd-92b2-155b4db81db6-ca-trust-extracted\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.815641 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.815620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afae54d8-af97-45fd-92b2-155b4db81db6-ca-trust-extracted\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.815641 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.815633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afae54d8-af97-45fd-92b2-155b4db81db6-registry-certificates\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.816262 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.816238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afae54d8-af97-45fd-92b2-155b4db81db6-trusted-ca\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.818852 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.818823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afae54d8-af97-45fd-92b2-155b4db81db6-image-registry-private-configuration\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.818852 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.818842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afae54d8-af97-45fd-92b2-155b4db81db6-installation-pull-secrets\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.818992 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.818833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23-cert\") pod \"ingress-canary-9pkl9\" (UID: \"6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23\") " pod="openshift-ingress-canary/ingress-canary-9pkl9"
Sep 18 19:57:13.838086 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.838060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-bound-sa-token\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.838332 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.838313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmsd9\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-kube-api-access-cmsd9\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:13.845714 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.845694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vg97\" (UniqueName: \"kubernetes.io/projected/6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23-kube-api-access-7vg97\") pod \"ingress-canary-9pkl9\" (UID: \"6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23\") " pod="openshift-ingress-canary/ingress-canary-9pkl9"
Sep 18 19:57:13.915971 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.915922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:13.916282 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.916026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9cb99c-d1fd-46f8-ae30-981161eaf053-config-volume\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:13.916282 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.916043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a9cb99c-d1fd-46f8-ae30-981161eaf053-tmp-dir\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:13.916282 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.916063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m74l\" (UniqueName: \"kubernetes.io/projected/6a9cb99c-d1fd-46f8-ae30-981161eaf053-kube-api-access-2m74l\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:13.916282 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.916066    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:57:13.916282 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:13.916118    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls podName:6a9cb99c-d1fd-46f8-ae30-981161eaf053 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:14.416104868 +0000 UTC m=+36.254568798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls") pod "dns-default-9h4s5" (UID: "6a9cb99c-d1fd-46f8-ae30-981161eaf053") : secret "dns-default-metrics-tls" not found
Sep 18 19:57:13.916444 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.916342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a9cb99c-d1fd-46f8-ae30-981161eaf053-tmp-dir\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:13.916874 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.916561    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a9cb99c-d1fd-46f8-ae30-981161eaf053-config-volume\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:13.931201 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.931178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m74l\" (UniqueName: \"kubernetes.io/projected/6a9cb99c-d1fd-46f8-ae30-981161eaf053-kube-api-access-2m74l\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:13.960839 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:13.960817    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9pkl9"
Sep 18 19:57:14.008679 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:14.008645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:14.168830 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:14.168654    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:14.299342 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:14.299306    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9pkl9"]
Sep 18 19:57:14.303624 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:57:14.303597    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aff7ff3_e6a6_4db7_8fc9_e21503f5ad23.slice/crio-dbc44faca4bc314c5ab6b16efcd5f950d6e8f08fab00c0336f22b554791bb0d1 WatchSource:0}: Error finding container dbc44faca4bc314c5ab6b16efcd5f950d6e8f08fab00c0336f22b554791bb0d1: Status 404 returned error can't find the container with id dbc44faca4bc314c5ab6b16efcd5f950d6e8f08fab00c0336f22b554791bb0d1
Sep 18 19:57:14.319070 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:14.319046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:14.319192 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:14.319176    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:57:14.319232 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:14.319194    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4d89d599-dqphx: secret "image-registry-tls" not found
Sep 18 19:57:14.319275 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:14.319255    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls podName:afae54d8-af97-45fd-92b2-155b4db81db6 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:15.319238007 +0000 UTC m=+37.157701954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls") pod "image-registry-d4d89d599-dqphx" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6") : secret "image-registry-tls" not found
Sep 18 19:57:14.420319 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:14.420292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:14.420464 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:14.420443    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:57:14.420517 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:14.420508    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls podName:6a9cb99c-d1fd-46f8-ae30-981161eaf053 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:15.420493659 +0000 UTC m=+37.258957589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls") pod "dns-default-9h4s5" (UID: "6a9cb99c-d1fd-46f8-ae30-981161eaf053") : secret "dns-default-metrics-tls" not found
Sep 18 19:57:14.632595 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:14.632505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9pkl9" event={"ID":"6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23","Type":"ContainerStarted","Data":"dbc44faca4bc314c5ab6b16efcd5f950d6e8f08fab00c0336f22b554791bb0d1"}
Sep 18 19:57:14.635190 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:14.635165    2578 generic.go:356] "Generic (PLEG): container finished" podID="2652ec9f-d178-4d1d-b37d-b0644b52b7b3" containerID="9f38a0621bb058385de898ebfd8373373baa875f009b027ebb43f409b820a8a3" exitCode=0
Sep 18 19:57:14.635310 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:14.635215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpkj8" event={"ID":"2652ec9f-d178-4d1d-b37d-b0644b52b7b3","Type":"ContainerDied","Data":"9f38a0621bb058385de898ebfd8373373baa875f009b027ebb43f409b820a8a3"}
Sep 18 19:57:15.312314 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.312245    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:57:15.312314 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.312276    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:57:15.313125 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.312437    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:15.315442 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.315391    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:57:15.315442 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.315394    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:57:15.315627 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.315498    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:57:15.316108 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.316078    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gb8gb\""
Sep 18 19:57:15.316445 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.316420    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:57:15.316445 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.316429    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6rwzk\""
Sep 18 19:57:15.327742 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.327510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:15.327742 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:15.327663    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:57:15.327742 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:15.327677    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4d89d599-dqphx: secret "image-registry-tls" not found
Sep 18 19:57:15.327742 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:15.327736    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls podName:afae54d8-af97-45fd-92b2-155b4db81db6 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:17.327719644 +0000 UTC m=+39.166183578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls") pod "image-registry-d4d89d599-dqphx" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6") : secret "image-registry-tls" not found
Sep 18 19:57:15.375171 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.375137    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv"]
Sep 18 19:57:15.398959 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.398925    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"]
Sep 18 19:57:15.399105 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.399012    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv"
Sep 18 19:57:15.402843 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.402824    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 19:57:15.402972 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.402824    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 19:57:15.402972 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.402859    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 19:57:15.402972 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.402830    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 19:57:15.403562 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.403544    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-g44xw\""
Sep 18 19:57:15.415123 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.415019    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv"]
Sep 18 19:57:15.415123 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.415051    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"]
Sep 18 19:57:15.415123 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.415072    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"]
Sep 18 19:57:15.415123 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.415112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.418475 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.418458    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 19:57:15.418570 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.418483    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 19:57:15.418570 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.418459    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 19:57:15.419087 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.419070    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 19:57:15.428384 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.428362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:15.428516 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:15.428501    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:57:15.428585 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:15.428565    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls podName:6a9cb99c-d1fd-46f8-ae30-981161eaf053 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:17.42854736 +0000 UTC m=+39.267011308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls") pod "dns-default-9h4s5" (UID: "6a9cb99c-d1fd-46f8-ae30-981161eaf053") : secret "dns-default-metrics-tls" not found
Sep 18 19:57:15.431895 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.431875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"
Sep 18 19:57:15.434507 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.434486    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"]
Sep 18 19:57:15.438409 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.438391    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 19:57:15.529319 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.529280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4dc2d945-f2c6-417a-9371-b35b439f2d9c-ca\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.529483 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.529347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e8a4a5fa-832c-4c0c-91a1-79d1305f6b02-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv\" (UID: \"e8a4a5fa-832c-4c0c-91a1-79d1305f6b02\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv"
Sep 18 19:57:15.529483 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.529380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x7jt\" (UniqueName: \"kubernetes.io/projected/e8a4a5fa-832c-4c0c-91a1-79d1305f6b02-kube-api-access-7x7jt\") pod \"managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv\" (UID: \"e8a4a5fa-832c-4c0c-91a1-79d1305f6b02\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv"
Sep 18 19:57:15.529483 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.529455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4dc2d945-f2c6-417a-9371-b35b439f2d9c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.529483 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.529482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl6jm\" (UniqueName: \"kubernetes.io/projected/4dc2d945-f2c6-417a-9371-b35b439f2d9c-kube-api-access-cl6jm\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.529837 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.529514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/459684ef-b6ea-4c42-b81d-cbcf1bd1c98e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fc4fc7d89-rp2fl\" (UID: \"459684ef-b6ea-4c42-b81d-cbcf1bd1c98e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"
Sep 18 19:57:15.529837 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.529628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4dc2d945-f2c6-417a-9371-b35b439f2d9c-hub\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.529837 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.529669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc2d945-f2c6-417a-9371-b35b439f2d9c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.529837 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.529699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4dc2d945-f2c6-417a-9371-b35b439f2d9c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.529837 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.529722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/459684ef-b6ea-4c42-b81d-cbcf1bd1c98e-tmp\") pod \"klusterlet-addon-workmgr-5fc4fc7d89-rp2fl\" (UID: \"459684ef-b6ea-4c42-b81d-cbcf1bd1c98e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"
Sep 18 19:57:15.529837 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.529747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-546cr\" (UniqueName: \"kubernetes.io/projected/459684ef-b6ea-4c42-b81d-cbcf1bd1c98e-kube-api-access-546cr\") pod \"klusterlet-addon-workmgr-5fc4fc7d89-rp2fl\" (UID: \"459684ef-b6ea-4c42-b81d-cbcf1bd1c98e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"
Sep 18 19:57:15.630553 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.630462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4dc2d945-f2c6-417a-9371-b35b439f2d9c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.630553 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.630510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/459684ef-b6ea-4c42-b81d-cbcf1bd1c98e-tmp\") pod \"klusterlet-addon-workmgr-5fc4fc7d89-rp2fl\" (UID: \"459684ef-b6ea-4c42-b81d-cbcf1bd1c98e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"
Sep 18 19:57:15.630769 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.630567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-546cr\" (UniqueName: \"kubernetes.io/projected/459684ef-b6ea-4c42-b81d-cbcf1bd1c98e-kube-api-access-546cr\") pod \"klusterlet-addon-workmgr-5fc4fc7d89-rp2fl\" (UID: \"459684ef-b6ea-4c42-b81d-cbcf1bd1c98e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"
Sep 18 19:57:15.630769 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.630597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4dc2d945-f2c6-417a-9371-b35b439f2d9c-ca\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.630983 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.630958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e8a4a5fa-832c-4c0c-91a1-79d1305f6b02-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv\" (UID: \"e8a4a5fa-832c-4c0c-91a1-79d1305f6b02\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv"
Sep 18 19:57:15.631053 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.630994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x7jt\" (UniqueName: \"kubernetes.io/projected/e8a4a5fa-832c-4c0c-91a1-79d1305f6b02-kube-api-access-7x7jt\") pod \"managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv\" (UID: \"e8a4a5fa-832c-4c0c-91a1-79d1305f6b02\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv"
Sep 18 19:57:15.631053 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.630998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/459684ef-b6ea-4c42-b81d-cbcf1bd1c98e-tmp\") pod \"klusterlet-addon-workmgr-5fc4fc7d89-rp2fl\" (UID: \"459684ef-b6ea-4c42-b81d-cbcf1bd1c98e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"
Sep 18 19:57:15.631152 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.631090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4dc2d945-f2c6-417a-9371-b35b439f2d9c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.631152 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.631140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl6jm\" (UniqueName: \"kubernetes.io/projected/4dc2d945-f2c6-417a-9371-b35b439f2d9c-kube-api-access-cl6jm\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.631264 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.631182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/459684ef-b6ea-4c42-b81d-cbcf1bd1c98e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fc4fc7d89-rp2fl\" (UID: \"459684ef-b6ea-4c42-b81d-cbcf1bd1c98e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"
Sep 18 19:57:15.631313 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.631262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4dc2d945-f2c6-417a-9371-b35b439f2d9c-hub\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.631313 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.631288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc2d945-f2c6-417a-9371-b35b439f2d9c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.632061 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.632035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc2d945-f2c6-417a-9371-b35b439f2d9c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.634883 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.634859    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4dc2d945-f2c6-417a-9371-b35b439f2d9c-ca\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.635466 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.635444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4dc2d945-f2c6-417a-9371-b35b439f2d9c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.635562 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.635463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4dc2d945-f2c6-417a-9371-b35b439f2d9c-hub\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.635562 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.635497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/459684ef-b6ea-4c42-b81d-cbcf1bd1c98e-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fc4fc7d89-rp2fl\" (UID: \"459684ef-b6ea-4c42-b81d-cbcf1bd1c98e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"
Sep 18 19:57:15.635678 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.635616    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e8a4a5fa-832c-4c0c-91a1-79d1305f6b02-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv\" (UID: \"e8a4a5fa-832c-4c0c-91a1-79d1305f6b02\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv"
Sep 18 19:57:15.636268 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.636239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4dc2d945-f2c6-417a-9371-b35b439f2d9c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.639917 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.639756    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-546cr\" (UniqueName: \"kubernetes.io/projected/459684ef-b6ea-4c42-b81d-cbcf1bd1c98e-kube-api-access-546cr\") pod \"klusterlet-addon-workmgr-5fc4fc7d89-rp2fl\" (UID: \"459684ef-b6ea-4c42-b81d-cbcf1bd1c98e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"
Sep 18 19:57:15.640055 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.640039    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl6jm\" (UniqueName: \"kubernetes.io/projected/4dc2d945-f2c6-417a-9371-b35b439f2d9c-kube-api-access-cl6jm\") pod \"cluster-proxy-proxy-agent-5cf4786ff7-bc98m\" (UID: \"4dc2d945-f2c6-417a-9371-b35b439f2d9c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.640436 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.640408    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x7jt\" (UniqueName: \"kubernetes.io/projected/e8a4a5fa-832c-4c0c-91a1-79d1305f6b02-kube-api-access-7x7jt\") pod \"managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv\" (UID: \"e8a4a5fa-832c-4c0c-91a1-79d1305f6b02\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv"
Sep 18 19:57:15.642213 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.642187    2578 generic.go:356] "Generic (PLEG): container finished" podID="2652ec9f-d178-4d1d-b37d-b0644b52b7b3" containerID="d10e08429d40c649dc3fa1203fe43de791c84fd07b55ee94e2306e6f2f98789c" exitCode=0
Sep 18 19:57:15.642317 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.642221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpkj8" event={"ID":"2652ec9f-d178-4d1d-b37d-b0644b52b7b3","Type":"ContainerDied","Data":"d10e08429d40c649dc3fa1203fe43de791c84fd07b55ee94e2306e6f2f98789c"}
Sep 18 19:57:15.720360 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.720336    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv"
Sep 18 19:57:15.728040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.728016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:57:15.761876 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:15.761853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"
Sep 18 19:57:16.019515 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:16.019330    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv"]
Sep 18 19:57:16.026815 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:16.026785    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"]
Sep 18 19:57:16.072656 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:16.072620    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"]
Sep 18 19:57:16.639460 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:57:16.639430    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8a4a5fa_832c_4c0c_91a1_79d1305f6b02.slice/crio-4ab7ee7fa3bc5cb0ee53d52d5e5ebaa2ce56db86f84fc0e420b23abe80f0f7da WatchSource:0}: Error finding container 4ab7ee7fa3bc5cb0ee53d52d5e5ebaa2ce56db86f84fc0e420b23abe80f0f7da: Status 404 returned error can't find the container with id 4ab7ee7fa3bc5cb0ee53d52d5e5ebaa2ce56db86f84fc0e420b23abe80f0f7da
Sep 18 19:57:16.640374 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:57:16.640345    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459684ef_b6ea_4c42_b81d_cbcf1bd1c98e.slice/crio-660b1a8f4e26d61ba0cf1fa07383c5b75611da086d2cf1785c73bb5c89c00ef5 WatchSource:0}: Error finding container 660b1a8f4e26d61ba0cf1fa07383c5b75611da086d2cf1785c73bb5c89c00ef5: Status 404 returned error can't find the container with id 660b1a8f4e26d61ba0cf1fa07383c5b75611da086d2cf1785c73bb5c89c00ef5
Sep 18 19:57:16.641184 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:57:16.641148    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4dc2d945_f2c6_417a_9371_b35b439f2d9c.slice/crio-f5712c3af8662f7cdf41debcac8c47322e119af0cb7ecb141d7ae251685fc43e WatchSource:0}: Error finding container f5712c3af8662f7cdf41debcac8c47322e119af0cb7ecb141d7ae251685fc43e: Status 404 returned error can't find the container with id f5712c3af8662f7cdf41debcac8c47322e119af0cb7ecb141d7ae251685fc43e
Sep 18 19:57:16.648173 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:16.648146    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hpkj8" event={"ID":"2652ec9f-d178-4d1d-b37d-b0644b52b7b3","Type":"ContainerStarted","Data":"00b700bcbc8073d8c7d4684535bdb533cbcfd92367adf476a2b022acb1d408e9"}
Sep 18 19:57:16.649556 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:16.649531    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl" event={"ID":"459684ef-b6ea-4c42-b81d-cbcf1bd1c98e","Type":"ContainerStarted","Data":"660b1a8f4e26d61ba0cf1fa07383c5b75611da086d2cf1785c73bb5c89c00ef5"}
Sep 18 19:57:16.650727 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:16.650700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m" event={"ID":"4dc2d945-f2c6-417a-9371-b35b439f2d9c","Type":"ContainerStarted","Data":"f5712c3af8662f7cdf41debcac8c47322e119af0cb7ecb141d7ae251685fc43e"}
Sep 18 19:57:16.651740 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:16.651721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv" event={"ID":"e8a4a5fa-832c-4c0c-91a1-79d1305f6b02","Type":"ContainerStarted","Data":"4ab7ee7fa3bc5cb0ee53d52d5e5ebaa2ce56db86f84fc0e420b23abe80f0f7da"}
Sep 18 19:57:16.669782 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:16.669740    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hpkj8" podStartSLOduration=6.446996417 podStartE2EDuration="36.669728543s" totalImagesPullingTime="30.222732126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:16.668635886 +0000 UTC m=+38.507099837" watchObservedRunningTime="2026-09-18 19:57:16.669728543 +0000 UTC m=+38.508192494"
Sep 18 19:57:16.826456 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:16.826418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:16.951334 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:16.951301    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:17.052869 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:17.052836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:17.130495 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:17.130460    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:17.344155 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:17.344117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:17.344324 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:17.344267    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:57:17.344324 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:17.344286    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4d89d599-dqphx: secret "image-registry-tls" not found
Sep 18 19:57:17.344408 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:17.344340    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls podName:afae54d8-af97-45fd-92b2-155b4db81db6 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:21.344324104 +0000 UTC m=+43.182788034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls") pod "image-registry-d4d89d599-dqphx" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6") : secret "image-registry-tls" not found
Sep 18 19:57:17.445105 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:17.445067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:17.445262 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:17.445229    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:57:17.445318 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:17.445296    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls podName:6a9cb99c-d1fd-46f8-ae30-981161eaf053 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:21.445275621 +0000 UTC m=+43.283739575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls") pod "dns-default-9h4s5" (UID: "6a9cb99c-d1fd-46f8-ae30-981161eaf053") : secret "dns-default-metrics-tls" not found
Sep 18 19:57:17.658131 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:17.658046    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9pkl9" event={"ID":"6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23","Type":"ContainerStarted","Data":"0ea9e489577f64b6d0c6f11cc897e77b21b3e7292f004aa2954dca7d0e39b7c4"}
Sep 18 19:57:17.958620 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:17.958513    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:18.030073 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:18.029987    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:18.144904 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:18.144870    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:18.232326 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:18.232018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:18.406509 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:18.406408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:18.504566 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:18.504452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:21.382346 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:21.382294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:21.382780 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:21.382472    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:57:21.382780 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:21.382494    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4d89d599-dqphx: secret "image-registry-tls" not found
Sep 18 19:57:21.382780 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:21.382563    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls podName:afae54d8-af97-45fd-92b2-155b4db81db6 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:29.382542792 +0000 UTC m=+51.221006729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls") pod "image-registry-d4d89d599-dqphx" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6") : secret "image-registry-tls" not found
Sep 18 19:57:21.483843 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:21.483804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:21.484030 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:21.483978    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:57:21.484094 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:21.484043    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls podName:6a9cb99c-d1fd-46f8-ae30-981161eaf053 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:29.484026192 +0000 UTC m=+51.322490123 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls") pod "dns-default-9h4s5" (UID: "6a9cb99c-d1fd-46f8-ae30-981161eaf053") : secret "dns-default-metrics-tls" not found
Sep 18 19:57:22.108510 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:22.108472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:22.210799 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:22.210765    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:22.411440 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:22.411354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:23.292104 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:23.292068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:23.405766 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:23.405733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:23.507163 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:23.507133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:24.678136 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:24.678091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl" event={"ID":"459684ef-b6ea-4c42-b81d-cbcf1bd1c98e","Type":"ContainerStarted","Data":"4448f9566cddd73865e0ca5af22e6d95b8ebe08f703ce2488eb8c3107acdeda5"}
Sep 18 19:57:24.678626 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:24.678245    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"
Sep 18 19:57:24.680216 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:24.680193    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl"
Sep 18 19:57:24.680538 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:24.680517    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m" event={"ID":"4dc2d945-f2c6-417a-9371-b35b439f2d9c","Type":"ContainerStarted","Data":"827049a3a7f3bb97bde2831fd5d5688042ebd4dfc6fe92c31f63738bdf112772"}
Sep 18 19:57:24.680623 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:24.680553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m" event={"ID":"4dc2d945-f2c6-417a-9371-b35b439f2d9c","Type":"ContainerStarted","Data":"c19d273a8e091fcfbd16b6be6a3f16d0cfc9cc5af12a1a06889f4d1bc373164b"}
Sep 18 19:57:24.680623 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:24.680571    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m" event={"ID":"4dc2d945-f2c6-417a-9371-b35b439f2d9c","Type":"ContainerStarted","Data":"95e88456c512d294e5f1cd80a2b49643b1f3950645420f4de02a5dae2a86f6c3"}
Sep 18 19:57:24.681998 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:24.681974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv" event={"ID":"e8a4a5fa-832c-4c0c-91a1-79d1305f6b02","Type":"ContainerStarted","Data":"b6dec6f4d4681f016fcf5445d191626ca1bb541e9e87f74e3f18a3dd2bde5147"}
Sep 18 19:57:24.695566 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:24.695526    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9pkl9" podStartSLOduration=9.094717878 podStartE2EDuration="11.695511214s" totalImagesPullingTime="2.600793336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:17.678826331 +0000 UTC m=+39.517290283" watchObservedRunningTime="2026-09-18 19:57:24.695511214 +0000 UTC m=+46.533975213"
Sep 18 19:57:24.695859 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:24.695839    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fc4fc7d89-rp2fl" podStartSLOduration=2.378093119 podStartE2EDuration="9.695831206s" totalImagesPullingTime="7.317738087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:24.694405312 +0000 UTC m=+46.532869264" watchObservedRunningTime="2026-09-18 19:57:24.695831206 +0000 UTC m=+46.534295155"
Sep 18 19:57:24.717385 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:24.717347    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m" podStartSLOduration=2.4182874610000002 podStartE2EDuration="9.717334281s" totalImagesPullingTime="7.29904682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:24.716302923 +0000 UTC m=+46.554766875" watchObservedRunningTime="2026-09-18 19:57:24.717334281 +0000 UTC m=+46.555798227"
Sep 18 19:57:24.754029 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:24.753988    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68b8cc7f74-zlrtv" podStartSLOduration=2.45402174 podStartE2EDuration="9.753975367s" totalImagesPullingTime="7.299953627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:24.753324511 +0000 UTC m=+46.591788465" watchObservedRunningTime="2026-09-18 19:57:24.753975367 +0000 UTC m=+46.592439313"
Sep 18 19:57:24.912810 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:24.912781    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:25.013357 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:25.013330    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:26.606618 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:26.606581    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:26.673563 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:26.673528    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:28.219405 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:28.219369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:29.099393 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:29.099358    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:29.183305 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:29.183216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:29.448516 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:29.448435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:29.448878 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:29.448545    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:57:29.448878 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:29.448558    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4d89d599-dqphx: secret "image-registry-tls" not found
Sep 18 19:57:29.448878 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:29.448612    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls podName:afae54d8-af97-45fd-92b2-155b4db81db6 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:45.448598314 +0000 UTC m=+67.287062244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls") pod "image-registry-d4d89d599-dqphx" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6") : secret "image-registry-tls" not found
Sep 18 19:57:29.549077 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:29.549039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:29.549183 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:29.549157    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:57:29.549227 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:29.549217    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls podName:6a9cb99c-d1fd-46f8-ae30-981161eaf053 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:45.549199767 +0000 UTC m=+67.387663698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls") pod "dns-default-9h4s5" (UID: "6a9cb99c-d1fd-46f8-ae30-981161eaf053") : secret "dns-default-metrics-tls" not found
Sep 18 19:57:34.150904 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:34.150861    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:34.281551 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:34.281517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:35.755717 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:35.755686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:35.852583 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:35.852550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:39.239091 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:39.239058    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:39.341474 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:39.341437    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:39.631177 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:39.631105    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kw5gp"
Sep 18 19:57:39.816943 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:39.816897    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:40.087020 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:40.086984    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:44.968951 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:44.968902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:57:44.971802 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:44.971781    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:57:44.979871 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:44.979856    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:57:44.979965 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:44.979915    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs podName:0519a79d-05b8-40fe-8d2a-d10b0bf04473 nodeName:}" failed. No retries permitted until 2026-09-18 19:58:48.979895102 +0000 UTC m=+130.818359031 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs") pod "network-metrics-daemon-6xbfv" (UID: "0519a79d-05b8-40fe-8d2a-d10b0bf04473") : secret "metrics-daemon-secret" not found
Sep 18 19:57:45.069993 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.069958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:57:45.072507 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.072487    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:57:45.083492 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.083469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea005810-7434-414a-8805-b1a2032e8108-original-pull-secret\") pod \"global-pull-secret-syncer-mshsc\" (UID: \"ea005810-7434-414a-8805-b1a2032e8108\") " pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:57:45.170335 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.170296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczdl\" (UniqueName: \"kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl\") pod \"network-check-target-bq6nj\" (UID: \"ac0567c9-870f-417a-a157-98253c4850b3\") " pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:45.173080 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.173060    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:57:45.183523 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.183502    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:57:45.193366 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.193336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sczdl\" (UniqueName: \"kubernetes.io/projected/ac0567c9-870f-417a-a157-98253c4850b3-kube-api-access-sczdl\") pod \"network-check-target-bq6nj\" (UID: \"ac0567c9-870f-417a-a157-98253c4850b3\") " pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:45.324026 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.323912    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mshsc"
Sep 18 19:57:45.339437 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.339416    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gb8gb\""
Sep 18 19:57:45.347400 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.347380    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:45.473601 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.473566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:57:45.473750 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:45.473720    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:57:45.473750 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:45.473740    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4d89d599-dqphx: secret "image-registry-tls" not found
Sep 18 19:57:45.473819 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:45.473810    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls podName:afae54d8-af97-45fd-92b2-155b4db81db6 nodeName:}" failed. No retries permitted until 2026-09-18 19:58:17.473792299 +0000 UTC m=+99.312256228 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls") pod "image-registry-d4d89d599-dqphx" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6") : secret "image-registry-tls" not found
Sep 18 19:57:45.574333 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.574260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:57:45.574465 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:45.574391    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:57:45.574465 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:57:45.574452    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls podName:6a9cb99c-d1fd-46f8-ae30-981161eaf053 nodeName:}" failed. No retries permitted until 2026-09-18 19:58:17.574436212 +0000 UTC m=+99.412900141 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls") pod "dns-default-9h4s5" (UID: "6a9cb99c-d1fd-46f8-ae30-981161eaf053") : secret "dns-default-metrics-tls" not found
Sep 18 19:57:45.594838 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.594813    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mshsc"]
Sep 18 19:57:45.598430 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:57:45.598406    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea005810_7434_414a_8805_b1a2032e8108.slice/crio-11036d29e6915d21b538187d2dee389137f80e417c08740fa22409ee340ab3b1 WatchSource:0}: Error finding container 11036d29e6915d21b538187d2dee389137f80e417c08740fa22409ee340ab3b1: Status 404 returned error can't find the container with id 11036d29e6915d21b538187d2dee389137f80e417c08740fa22409ee340ab3b1
Sep 18 19:57:45.622133 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.622111    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bq6nj"]
Sep 18 19:57:45.625761 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:57:45.625742    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0567c9_870f_417a_a157_98253c4850b3.slice/crio-e5b456ac2c7ab2d641d51550f090544593cd20a14108aa0187974e1121ecad54 WatchSource:0}: Error finding container e5b456ac2c7ab2d641d51550f090544593cd20a14108aa0187974e1121ecad54: Status 404 returned error can't find the container with id e5b456ac2c7ab2d641d51550f090544593cd20a14108aa0187974e1121ecad54
Sep 18 19:57:45.747895 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.747858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bq6nj" event={"ID":"ac0567c9-870f-417a-a157-98253c4850b3","Type":"ContainerStarted","Data":"e5b456ac2c7ab2d641d51550f090544593cd20a14108aa0187974e1121ecad54"}
Sep 18 19:57:45.748952 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:45.748915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mshsc" event={"ID":"ea005810-7434-414a-8805-b1a2032e8108","Type":"ContainerStarted","Data":"11036d29e6915d21b538187d2dee389137f80e417c08740fa22409ee340ab3b1"}
Sep 18 19:57:50.768126 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:50.768084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mshsc" event={"ID":"ea005810-7434-414a-8805-b1a2032e8108","Type":"ContainerStarted","Data":"f83c4629f21978fb2c81fe9caa6fe83fed7eb7f997de1fe1b7f466eab925f4a6"}
Sep 18 19:57:50.769607 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:50.769581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bq6nj" event={"ID":"ac0567c9-870f-417a-a157-98253c4850b3","Type":"ContainerStarted","Data":"0159770efad97f1e80333a22d5f03c9f2c747f747d1ba037f32a55bdb7c4d39e"}
Sep 18 19:57:50.769734 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:50.769717    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:57:50.784778 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:50.784724    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mshsc" podStartSLOduration=66.218708737 podStartE2EDuration="1m10.784708395s" totalImagesPullingTime="4.565999658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:50.784457871 +0000 UTC m=+72.622921825" watchObservedRunningTime="2026-09-18 19:57:50.784708395 +0000 UTC m=+72.623172347"
Sep 18 19:57:50.799607 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:50.799568    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bq6nj" podStartSLOduration=66.265613635 podStartE2EDuration="1m10.799555132s" totalImagesPullingTime="4.533941497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:50.79869422 +0000 UTC m=+72.637158171" watchObservedRunningTime="2026-09-18 19:57:50.799555132 +0000 UTC m=+72.638019088"
Sep 18 19:57:55.965374 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:55.965267    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:57.082621 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:57:57.082575    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:01.159896 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:01.159864    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:01.223854 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:01.223825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:01.293174 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:01.293147    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:17.505507 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:17.505465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls\") pod \"image-registry-d4d89d599-dqphx\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") " pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:58:17.505980 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:17.505624    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:58:17.505980 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:17.505646    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4d89d599-dqphx: secret "image-registry-tls" not found
Sep 18 19:58:17.505980 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:17.505705    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls podName:afae54d8-af97-45fd-92b2-155b4db81db6 nodeName:}" failed. No retries permitted until 2026-09-18 19:59:21.505688414 +0000 UTC m=+163.344152344 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls") pod "image-registry-d4d89d599-dqphx" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6") : secret "image-registry-tls" not found
Sep 18 19:58:17.606090 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:17.606057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:58:17.606248 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:17.606227    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:58:17.606334 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:17.606322    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls podName:6a9cb99c-d1fd-46f8-ae30-981161eaf053 nodeName:}" failed. No retries permitted until 2026-09-18 19:59:21.606299193 +0000 UTC m=+163.444763140 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls") pod "dns-default-9h4s5" (UID: "6a9cb99c-d1fd-46f8-ae30-981161eaf053") : secret "dns-default-metrics-tls" not found
Sep 18 19:58:20.703188 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:20.703155    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-hrwj8"]
Sep 18 19:58:20.704904 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:20.704889    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hrwj8"
Sep 18 19:58:20.708604 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:20.708579    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wnkdz\""
Sep 18 19:58:20.736887 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:20.736856    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-hrwj8"]
Sep 18 19:58:20.829085 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:20.829046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f27gr\" (UniqueName: \"kubernetes.io/projected/562fd9af-b875-4566-a33c-f83eb03757b2-kube-api-access-f27gr\") pod \"network-check-source-7cbc45fddc-hrwj8\" (UID: \"562fd9af-b875-4566-a33c-f83eb03757b2\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hrwj8"
Sep 18 19:58:20.868283 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:20.868250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:20.929408 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:20.929376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27gr\" (UniqueName: \"kubernetes.io/projected/562fd9af-b875-4566-a33c-f83eb03757b2-kube-api-access-f27gr\") pod \"network-check-source-7cbc45fddc-hrwj8\" (UID: \"562fd9af-b875-4566-a33c-f83eb03757b2\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hrwj8"
Sep 18 19:58:20.938124 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:20.938097    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f27gr\" (UniqueName: \"kubernetes.io/projected/562fd9af-b875-4566-a33c-f83eb03757b2-kube-api-access-f27gr\") pod \"network-check-source-7cbc45fddc-hrwj8\" (UID: \"562fd9af-b875-4566-a33c-f83eb03757b2\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hrwj8"
Sep 18 19:58:20.970133 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:20.970102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:21.013154 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:21.013128    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hrwj8"
Sep 18 19:58:21.280722 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:21.280655    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-hrwj8"]
Sep 18 19:58:21.284059 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:58:21.284030    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562fd9af_b875_4566_a33c_f83eb03757b2.slice/crio-765a67c092e3974f3881ac17e1710fc429da4e688fbacc39fc652c1546124a59 WatchSource:0}: Error finding container 765a67c092e3974f3881ac17e1710fc429da4e688fbacc39fc652c1546124a59: Status 404 returned error can't find the container with id 765a67c092e3974f3881ac17e1710fc429da4e688fbacc39fc652c1546124a59
Sep 18 19:58:21.775230 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:21.775198    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bq6nj"
Sep 18 19:58:21.860106 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:21.860071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hrwj8" event={"ID":"562fd9af-b875-4566-a33c-f83eb03757b2","Type":"ContainerStarted","Data":"7333a1ba8b6bacde756045530b2a0f13c9084500209b398e8baa1e21a09667c2"}
Sep 18 19:58:21.860281 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:21.860160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hrwj8" event={"ID":"562fd9af-b875-4566-a33c-f83eb03757b2","Type":"ContainerStarted","Data":"765a67c092e3974f3881ac17e1710fc429da4e688fbacc39fc652c1546124a59"}
Sep 18 19:58:21.877575 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:21.877523    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-hrwj8" podStartSLOduration=1.877507387 podStartE2EDuration="1.877507387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:21.875830454 +0000 UTC m=+103.714294409" watchObservedRunningTime="2026-09-18 19:58:21.877507387 +0000 UTC m=+103.715971338"
Sep 18 19:58:23.108024 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:23.107988    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:23.181502 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:23.181471    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:23.911297 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:23.911267    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wxpxl"]
Sep 18 19:58:23.914921 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:23.914905    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:23.917249 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:23.917215    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:58:23.917249 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:23.917235    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:58:23.917425 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:23.917267    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:58:23.917549 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:23.917533    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:58:23.918036 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:23.918022    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dnrhk\""
Sep 18 19:58:23.932374 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:23.932352    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wxpxl"]
Sep 18 19:58:23.949829 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:23.949805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:24.049547 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.049513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.049547 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.049546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.049770 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.049575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/620c86f5-7746-491c-bc47-128d734e350f-crio-socket\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.049770 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.049590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/620c86f5-7746-491c-bc47-128d734e350f-data-volume\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.049770 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.049681    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k79gh\" (UniqueName: \"kubernetes.io/projected/620c86f5-7746-491c-bc47-128d734e350f-kube-api-access-k79gh\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.057240 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.057213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:24.150154 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.150125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.150154 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.150154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.150577 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.150178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/620c86f5-7746-491c-bc47-128d734e350f-crio-socket\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.150577 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.150195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/620c86f5-7746-491c-bc47-128d734e350f-data-volume\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.150577 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.150216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k79gh\" (UniqueName: \"kubernetes.io/projected/620c86f5-7746-491c-bc47-128d734e350f-kube-api-access-k79gh\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.150577 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:24.150263    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:58:24.150577 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:24.150283    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:24.150577 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.150285    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/620c86f5-7746-491c-bc47-128d734e350f-crio-socket\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.150577 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:24.150344    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls podName:620c86f5-7746-491c-bc47-128d734e350f nodeName:}" failed. No retries permitted until 2026-09-18 19:58:24.650323429 +0000 UTC m=+106.488787376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxpxl" (UID: "620c86f5-7746-491c-bc47-128d734e350f") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:24.150577 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:24.150363    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm podName:620c86f5-7746-491c-bc47-128d734e350f nodeName:}" failed. No retries permitted until 2026-09-18 19:58:24.650353966 +0000 UTC m=+106.488817896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wxpxl" (UID: "620c86f5-7746-491c-bc47-128d734e350f") : configmap "kube-rbac-proxy" not found
Sep 18 19:58:24.150577 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.150483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/620c86f5-7746-491c-bc47-128d734e350f-data-volume\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.159857 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.159829    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k79gh\" (UniqueName: \"kubernetes.io/projected/620c86f5-7746-491c-bc47-128d734e350f-kube-api-access-k79gh\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.654073 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.654031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.654073 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:24.654075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:24.654342 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:24.654164    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:58:24.654342 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:24.654214    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:24.654342 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:24.654256    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm podName:620c86f5-7746-491c-bc47-128d734e350f nodeName:}" failed. No retries permitted until 2026-09-18 19:58:25.654234703 +0000 UTC m=+107.492698636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wxpxl" (UID: "620c86f5-7746-491c-bc47-128d734e350f") : configmap "kube-rbac-proxy" not found
Sep 18 19:58:24.654342 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:24.654275    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls podName:620c86f5-7746-491c-bc47-128d734e350f nodeName:}" failed. No retries permitted until 2026-09-18 19:58:25.654266814 +0000 UTC m=+107.492730743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxpxl" (UID: "620c86f5-7746-491c-bc47-128d734e350f") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:25.494217 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:25.494182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:25.580003 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:25.579973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:25.661487 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:25.661454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:25.661487 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:25.661490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:25.661666 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:25.661573    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:58:25.661666 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:25.661632    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm podName:620c86f5-7746-491c-bc47-128d734e350f nodeName:}" failed. No retries permitted until 2026-09-18 19:58:27.661616299 +0000 UTC m=+109.500080228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wxpxl" (UID: "620c86f5-7746-491c-bc47-128d734e350f") : configmap "kube-rbac-proxy" not found
Sep 18 19:58:25.661666 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:25.661649    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:25.661768 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:25.661694    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls podName:620c86f5-7746-491c-bc47-128d734e350f nodeName:}" failed. No retries permitted until 2026-09-18 19:58:27.661677537 +0000 UTC m=+109.500141467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxpxl" (UID: "620c86f5-7746-491c-bc47-128d734e350f") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:25.821508 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:25.821439    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fhfr4_a5470bc7-569a-490e-8ac3-e63b5738b859/dns-node-resolver/0.log"
Sep 18 19:58:25.923675 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:25.923643    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:26.002857 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:26.002827    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:26.271872 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:26.271831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:26.391204 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:26.391169    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:26.514143 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:26.514111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:26.613588 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:26.613503    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:26.622752 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:26.622732    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7shwk_1685cb44-5bb4-493c-b414-cb4e2ec0ab2f/node-ca/0.log"
Sep 18 19:58:27.421343 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:27.421312    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9pkl9_6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23/serve-healthcheck-canary/0.log"
Sep 18 19:58:27.563668 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:27.563644    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:27.646723 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:27.646688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:27.679753 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:27.679697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:27.679753 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:27.679727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:27.679886 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:27.679820    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:58:27.679886 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:27.679847    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:27.679886 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:27.679879    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm podName:620c86f5-7746-491c-bc47-128d734e350f nodeName:}" failed. No retries permitted until 2026-09-18 19:58:31.679863785 +0000 UTC m=+113.518327715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wxpxl" (UID: "620c86f5-7746-491c-bc47-128d734e350f") : configmap "kube-rbac-proxy" not found
Sep 18 19:58:27.680093 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:27.679895    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls podName:620c86f5-7746-491c-bc47-128d734e350f nodeName:}" failed. No retries permitted until 2026-09-18 19:58:31.67988392 +0000 UTC m=+113.518347850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxpxl" (UID: "620c86f5-7746-491c-bc47-128d734e350f") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:27.823327 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:27.823292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:27.892924 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:27.892874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:28.869343 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:28.869310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:29.785106 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:29.785075    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:29.913601 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:29.913559    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:30.000990 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:30.000959    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:30.073524 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:30.073436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:30.143560 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:30.143521    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:30.332823 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:30.332745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:30.521090 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:30.521052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:31.462482 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:31.462449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:31.711639 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:31.711594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:31.711825 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:31.711696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:31.711825 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:31.711742    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:31.711825 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:31.711758    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:58:31.711825 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:31.711806    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm podName:620c86f5-7746-491c-bc47-128d734e350f nodeName:}" failed. No retries permitted until 2026-09-18 19:58:39.711793424 +0000 UTC m=+121.550257354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wxpxl" (UID: "620c86f5-7746-491c-bc47-128d734e350f") : configmap "kube-rbac-proxy" not found
Sep 18 19:58:31.711825 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:31.711818    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls podName:620c86f5-7746-491c-bc47-128d734e350f nodeName:}" failed. No retries permitted until 2026-09-18 19:58:39.711812486 +0000 UTC m=+121.550276416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxpxl" (UID: "620c86f5-7746-491c-bc47-128d734e350f") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:31.881101 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:31.881066    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:32.076607 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:32.076580    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:32.268798 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:32.268756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:32.467337 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:32.467300    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:33.189690 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:33.189659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:33.498504 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:33.498472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:33.603643 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:33.603616    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:33.709454 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:33.709422    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:34.085046 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:34.085014    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:34.303365 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:34.303305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:34.400258 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:34.400170    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:34.732295 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:34.732267    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:35.511064 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:35.511033    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:35.775571 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:35.775486    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:36.215922 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:36.215890    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:36.438417 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:36.438365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:36.905819 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:36.905785    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:37.122677 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:37.122649    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:37.296596 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:37.296561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:37.680003 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:37.679900    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:38.073343 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:38.073314    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:38.472288 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:38.472255    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:38.666225 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:38.666175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:39.072228 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:39.072196    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:39.664945 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:39.664912    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:39.772894 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:39.772859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:39.772894 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:39.772897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:39.773113 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:39.773034    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:39.773113 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:58:39.773101    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls podName:620c86f5-7746-491c-bc47-128d734e350f nodeName:}" failed. No retries permitted until 2026-09-18 19:58:55.77308498 +0000 UTC m=+137.611548927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wxpxl" (UID: "620c86f5-7746-491c-bc47-128d734e350f") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:58:39.773439 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:39.773421    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/620c86f5-7746-491c-bc47-128d734e350f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:40.126273 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:40.126233    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:40.287384 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:40.287350    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:40.470556 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:40.470522    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:40.973731 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:40.973696    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:41.617069 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:41.617039    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:41.718113 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:41.718083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:45.729775 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:45.729722    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m" podUID="4dc2d945-f2c6-417a-9371-b35b439f2d9c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 19:58:46.549646 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:46.549606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:46.648438 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:46.648407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:46.840064 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:46.839970    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:46.996974 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:46.996928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:47.118157 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:47.118079    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:47.204569 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:47.204527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:48.224644 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:48.224611    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:48.403989 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:48.403954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:48.603997 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:48.603901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:48.818669 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:48.818638    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:48.903744 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:48.903661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:49.041034 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:49.040999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:58:49.043313 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:49.043296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0519a79d-05b8-40fe-8d2a-d10b0bf04473-metrics-certs\") pod \"network-metrics-daemon-6xbfv\" (UID: \"0519a79d-05b8-40fe-8d2a-d10b0bf04473\") " pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:58:49.078363 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:49.078336    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:49.234028 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:49.234000    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6rwzk\""
Sep 18 19:58:49.241680 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:49.241659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbfv"
Sep 18 19:58:49.516101 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:49.516078    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xbfv"]
Sep 18 19:58:49.517963 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:58:49.517916    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0519a79d_05b8_40fe_8d2a_d10b0bf04473.slice/crio-bb127cab1be99eae1181880b0c62e2d2ffea746bcb8124434022a3196f44d36a WatchSource:0}: Error finding container bb127cab1be99eae1181880b0c62e2d2ffea746bcb8124434022a3196f44d36a: Status 404 returned error can't find the container with id bb127cab1be99eae1181880b0c62e2d2ffea746bcb8124434022a3196f44d36a
Sep 18 19:58:49.942225 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:49.942135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xbfv" event={"ID":"0519a79d-05b8-40fe-8d2a-d10b0bf04473","Type":"ContainerStarted","Data":"bb127cab1be99eae1181880b0c62e2d2ffea746bcb8124434022a3196f44d36a"}
Sep 18 19:58:50.946634 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:50.946541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xbfv" event={"ID":"0519a79d-05b8-40fe-8d2a-d10b0bf04473","Type":"ContainerStarted","Data":"c2b4c786d62add880c336cccf7abe9206a04f8ab4e8d2f66770cd23a2be8fdac"}
Sep 18 19:58:50.946634 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:50.946624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xbfv" event={"ID":"0519a79d-05b8-40fe-8d2a-d10b0bf04473","Type":"ContainerStarted","Data":"4a1c766093bf561a7293b7ccb73f7b40bc445c992d5952b2478728a045b45161"}
Sep 18 19:58:50.976334 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:50.976285    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6xbfv" podStartSLOduration=129.951874484 podStartE2EDuration="2m10.976270868s" totalImagesPullingTime="1.024396384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:50.975638606 +0000 UTC m=+132.814102559" watchObservedRunningTime="2026-09-18 19:58:50.976270868 +0000 UTC m=+132.814734820"
Sep 18 19:58:52.504547 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:52.504510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:52.821425 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:52.821345    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:53.036162 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:53.036128    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:54.526013 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:54.525980    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:54.614201 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:54.614165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:55.502346 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:55.502310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:55.600979 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:55.600945    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:55.700529 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:55.700499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:55.729720 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:55.729686    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m" podUID="4dc2d945-f2c6-417a-9371-b35b439f2d9c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 19:58:55.792788 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:55.792710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:55.794959 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:55.794915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/620c86f5-7746-491c-bc47-128d734e350f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wxpxl\" (UID: \"620c86f5-7746-491c-bc47-128d734e350f\") " pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:55.871819 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:55.871780    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:56.026114 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:56.026088    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dnrhk\""
Sep 18 19:58:56.033939 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:56.033915    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wxpxl"
Sep 18 19:58:56.325486 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:56.325458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wxpxl"]
Sep 18 19:58:56.326082 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:58:56.326056    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620c86f5_7746_491c_bc47_128d734e350f.slice/crio-94f69718151fc873aa01633b1fd925924f211ef9e728763f589a176b516082b8 WatchSource:0}: Error finding container 94f69718151fc873aa01633b1fd925924f211ef9e728763f589a176b516082b8: Status 404 returned error can't find the container with id 94f69718151fc873aa01633b1fd925924f211ef9e728763f589a176b516082b8
Sep 18 19:58:56.966617 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:56.966593    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wxpxl" event={"ID":"620c86f5-7746-491c-bc47-128d734e350f","Type":"ContainerStarted","Data":"153da55756b60df3781f1dc6d2999c615f175258b1cde715a6ee405a27ff264e"}
Sep 18 19:58:56.966894 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:56.966643    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wxpxl" event={"ID":"620c86f5-7746-491c-bc47-128d734e350f","Type":"ContainerStarted","Data":"94f69718151fc873aa01633b1fd925924f211ef9e728763f589a176b516082b8"}
Sep 18 19:58:57.972652 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:57.972608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wxpxl" event={"ID":"620c86f5-7746-491c-bc47-128d734e350f","Type":"ContainerStarted","Data":"84c440e852521c072ae039c0f80cce25686c6a8dddf139b2bcb4b7df6c9ce630"}
Sep 18 19:58:58.419839 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:58.419801    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:58.519647 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:58.519559    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:58.713642 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:58.713611    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:58.819174 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:58.819093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:58.978434 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:58.978402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wxpxl" event={"ID":"620c86f5-7746-491c-bc47-128d734e350f","Type":"ContainerStarted","Data":"beda74ceee5cbdf1522d6c5839469f7e3ac3d1f68c957abb8ed9fd66f47585f8"}
Sep 18 19:58:59.002892 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:59.002837    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wxpxl" podStartSLOduration=34.198388767 podStartE2EDuration="36.002821483s" totalImagesPullingTime="1.804432716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:58:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:59.002060442 +0000 UTC m=+140.840524429" watchObservedRunningTime="2026-09-18 19:58:59.002821483 +0000 UTC m=+140.841285434"
Sep 18 19:58:59.222532 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:58:59.222499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:01.763404 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:01.763363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:01.834996 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:01.834962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:02.021167 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:02.021092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:02.092039 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:02.092008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:02.416397 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:02.416303    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:02.487569 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:02.487533    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:05.729293 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:05.729256    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m" podUID="4dc2d945-f2c6-417a-9371-b35b439f2d9c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 19:59:05.729645 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:05.729334    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m"
Sep 18 19:59:05.729885 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:05.729854    2578 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"827049a3a7f3bb97bde2831fd5d5688042ebd4dfc6fe92c31f63738bdf112772"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 18 19:59:05.729924 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:05.729910    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m" podUID="4dc2d945-f2c6-417a-9371-b35b439f2d9c" containerName="service-proxy" containerID="cri-o://827049a3a7f3bb97bde2831fd5d5688042ebd4dfc6fe92c31f63738bdf112772" gracePeriod=30
Sep 18 19:59:06.000664 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:06.000585    2578 generic.go:356] "Generic (PLEG): container finished" podID="4dc2d945-f2c6-417a-9371-b35b439f2d9c" containerID="827049a3a7f3bb97bde2831fd5d5688042ebd4dfc6fe92c31f63738bdf112772" exitCode=2
Sep 18 19:59:06.000664 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:06.000641    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m" event={"ID":"4dc2d945-f2c6-417a-9371-b35b439f2d9c","Type":"ContainerDied","Data":"827049a3a7f3bb97bde2831fd5d5688042ebd4dfc6fe92c31f63738bdf112772"}
Sep 18 19:59:06.000820 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:06.000681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cf4786ff7-bc98m" event={"ID":"4dc2d945-f2c6-417a-9371-b35b439f2d9c","Type":"ContainerStarted","Data":"148a8fe86cfad53a3122c44281dee867d8f11d15f452163ed443c169fe248ea2"}
Sep 18 19:59:06.248273 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:06.248244    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:06.395301 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:06.395230    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:08.338444 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:08.338410    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:08.454188 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:08.454150    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:08.551235 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:08.551201    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:10.636841 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:10.636804    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:10.740478 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:10.740447    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:11.590595 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:11.590565    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:11.695766 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:11.695729    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:12.099330 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:12.099297    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:12.331378 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:12.331340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:12.422832 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:12.422750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:12.522833 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:12.522803    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:12.696344 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:12.696270    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:12.800349 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:12.800313    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:14.406915 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:14.406878    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:14.466972 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:14.466928    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d4d89d599-dqphx"]
Sep 18 19:59:14.467155 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:59:14.467138    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-d4d89d599-dqphx" podUID="afae54d8-af97-45fd-92b2-155b4db81db6"
Sep 18 19:59:14.654810 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:14.654778    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:14.836692 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:14.836660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:15.026464 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.026429    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:59:15.032674 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.032653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:59:15.097545 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.097454    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7pfwf"]
Sep 18 19:59:15.100393 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.100375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.114160 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.114141    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:59:15.114268 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.114233    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:59:15.114439 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.114424    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c6mwt\""
Sep 18 19:59:15.114579 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.114564    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:59:15.115187 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.115173    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:59:15.117352 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.117335    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:59:15.129292 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.129272    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:59:15.137005 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.136987    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afae54d8-af97-45fd-92b2-155b4db81db6-trusted-ca\") pod \"afae54d8-af97-45fd-92b2-155b4db81db6\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") "
Sep 18 19:59:15.137065 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.137013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-bound-sa-token\") pod \"afae54d8-af97-45fd-92b2-155b4db81db6\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") "
Sep 18 19:59:15.137065 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.137042    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afae54d8-af97-45fd-92b2-155b4db81db6-image-registry-private-configuration\") pod \"afae54d8-af97-45fd-92b2-155b4db81db6\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") "
Sep 18 19:59:15.137065 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.137062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afae54d8-af97-45fd-92b2-155b4db81db6-ca-trust-extracted\") pod \"afae54d8-af97-45fd-92b2-155b4db81db6\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") "
Sep 18 19:59:15.137187 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.137089    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afae54d8-af97-45fd-92b2-155b4db81db6-registry-certificates\") pod \"afae54d8-af97-45fd-92b2-155b4db81db6\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") "
Sep 18 19:59:15.137187 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.137109    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afae54d8-af97-45fd-92b2-155b4db81db6-installation-pull-secrets\") pod \"afae54d8-af97-45fd-92b2-155b4db81db6\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") "
Sep 18 19:59:15.137187 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.137158    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmsd9\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-kube-api-access-cmsd9\") pod \"afae54d8-af97-45fd-92b2-155b4db81db6\" (UID: \"afae54d8-af97-45fd-92b2-155b4db81db6\") "
Sep 18 19:59:15.137447 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.137385    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afae54d8-af97-45fd-92b2-155b4db81db6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "afae54d8-af97-45fd-92b2-155b4db81db6" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:15.137560 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.137529    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afae54d8-af97-45fd-92b2-155b4db81db6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "afae54d8-af97-45fd-92b2-155b4db81db6" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:59:15.137697 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.137671    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afae54d8-af97-45fd-92b2-155b4db81db6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "afae54d8-af97-45fd-92b2-155b4db81db6" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:59:15.138901 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.138876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:15.139663 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.139621    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "afae54d8-af97-45fd-92b2-155b4db81db6" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:59:15.139760 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.139730    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-kube-api-access-cmsd9" (OuterVolumeSpecName: "kube-api-access-cmsd9") pod "afae54d8-af97-45fd-92b2-155b4db81db6" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6"). InnerVolumeSpecName "kube-api-access-cmsd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:59:15.139760 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.139745    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afae54d8-af97-45fd-92b2-155b4db81db6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "afae54d8-af97-45fd-92b2-155b4db81db6" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:59:15.139874 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.139793    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afae54d8-af97-45fd-92b2-155b4db81db6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "afae54d8-af97-45fd-92b2-155b4db81db6" (UID: "afae54d8-af97-45fd-92b2-155b4db81db6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:59:15.238460 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-wtmp\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.238460 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-metrics-client-ca\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.238694 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-root\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.238694 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-tls\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.238694 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-textfile\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.238694 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238659    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.238869 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5f42\" (UniqueName: \"kubernetes.io/projected/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-kube-api-access-r5f42\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.238869 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-sys\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.238869 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.238869 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238834    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/afae54d8-af97-45fd-92b2-155b4db81db6-registry-certificates\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 19:59:15.238869 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238851    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/afae54d8-af97-45fd-92b2-155b4db81db6-installation-pull-secrets\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 19:59:15.238869 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238864    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmsd9\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-kube-api-access-cmsd9\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 19:59:15.239073 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238874    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/afae54d8-af97-45fd-92b2-155b4db81db6-trusted-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 19:59:15.239073 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238883    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-bound-sa-token\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 19:59:15.239073 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238891    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/afae54d8-af97-45fd-92b2-155b4db81db6-image-registry-private-configuration\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 19:59:15.239073 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.238900    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/afae54d8-af97-45fd-92b2-155b4db81db6-ca-trust-extracted\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 19:59:15.321289 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.321252    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:15.340132 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-wtmp\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340284 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-metrics-client-ca\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340284 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-root\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340284 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-tls\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340284 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-textfile\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340284 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-root\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340284 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340570 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5f42\" (UniqueName: \"kubernetes.io/projected/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-kube-api-access-r5f42\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340570 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-sys\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340570 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340570 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-wtmp\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340570 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-sys\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340758 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-textfile\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340823 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-metrics-client-ca\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.340951 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.340913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.342668 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.342645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.342756 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.342699    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-node-exporter-tls\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.360766 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.360714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5f42\" (UniqueName: \"kubernetes.io/projected/b0a00281-39c2-4082-a8aa-8f5f4703d1ab-kube-api-access-r5f42\") pod \"node-exporter-7pfwf\" (UID: \"b0a00281-39c2-4082-a8aa-8f5f4703d1ab\") " pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.411851 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.411820    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7pfwf"
Sep 18 19:59:15.423244 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:59:15.423222    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a00281_39c2_4082_a8aa_8f5f4703d1ab.slice/crio-24816e0689d66f914485c667364deedb3867afb85c4aeeccfa79ea3e456878df WatchSource:0}: Error finding container 24816e0689d66f914485c667364deedb3867afb85c4aeeccfa79ea3e456878df: Status 404 returned error can't find the container with id 24816e0689d66f914485c667364deedb3867afb85c4aeeccfa79ea3e456878df
Sep 18 19:59:15.510246 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.510213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:15.648285 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.648197    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:15.755908 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.755877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:15.963454 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:15.963422    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:16.031000 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:16.030965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7pfwf" event={"ID":"b0a00281-39c2-4082-a8aa-8f5f4703d1ab","Type":"ContainerStarted","Data":"24816e0689d66f914485c667364deedb3867afb85c4aeeccfa79ea3e456878df"}
Sep 18 19:59:16.031000 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:16.031000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d4d89d599-dqphx"
Sep 18 19:59:16.100060 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:16.100027    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d4d89d599-dqphx"]
Sep 18 19:59:16.105831 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:16.105801    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d4d89d599-dqphx"]
Sep 18 19:59:16.187439 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:16.187407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:16.246851 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:16.246782    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/afae54d8-af97-45fd-92b2-155b4db81db6-registry-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 19:59:16.319298 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:16.319262    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afae54d8-af97-45fd-92b2-155b4db81db6" path="/var/lib/kubelet/pods/afae54d8-af97-45fd-92b2-155b4db81db6/volumes"
Sep 18 19:59:16.684300 ip-10-0-132-172 kubenswrapper[2578]: E0918 19:59:16.684264    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9h4s5" podUID="6a9cb99c-d1fd-46f8-ae30-981161eaf053"
Sep 18 19:59:17.035470 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:17.035442    2578 generic.go:356] "Generic (PLEG): container finished" podID="b0a00281-39c2-4082-a8aa-8f5f4703d1ab" containerID="7ebcf42460c5f38f6fd034b3960610b51fedf791d228a55e14448d07e9f8d8df" exitCode=0
Sep 18 19:59:17.035623 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:17.035528    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7pfwf" event={"ID":"b0a00281-39c2-4082-a8aa-8f5f4703d1ab","Type":"ContainerDied","Data":"7ebcf42460c5f38f6fd034b3960610b51fedf791d228a55e14448d07e9f8d8df"}
Sep 18 19:59:17.035623 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:17.035596    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:59:17.709141 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:17.709059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:17.811268 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:17.811231    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:17.914576 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:17.914548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:18.041166 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:18.041129    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7pfwf" event={"ID":"b0a00281-39c2-4082-a8aa-8f5f4703d1ab","Type":"ContainerStarted","Data":"5fde8461108c4b9aaf0887c90ef836d5a72979fc92055578db98a9e017377c13"}
Sep 18 19:59:18.041330 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:18.041188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7pfwf" event={"ID":"b0a00281-39c2-4082-a8aa-8f5f4703d1ab","Type":"ContainerStarted","Data":"6a2449cb80a416cde8f502530427ff786180993c1ea289b872ed939d021f465e"}
Sep 18 19:59:18.487649 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:18.487609    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:19.054216 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:19.054145    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:19.154371 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:19.154334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:19.375121 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:19.375038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:20.049192 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:20.049156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:20.190244 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:20.190212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:20.288263 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:20.288229    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:21.027836 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:21.027807    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:21.687467 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:21.687428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:59:21.689765 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:21.689743    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a9cb99c-d1fd-46f8-ae30-981161eaf053-metrics-tls\") pod \"dns-default-9h4s5\" (UID: \"6a9cb99c-d1fd-46f8-ae30-981161eaf053\") " pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:59:21.839149 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:21.839118    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r2wjd\""
Sep 18 19:59:21.847311 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:21.847287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:59:22.120040 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:22.119912    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7pfwf" podStartSLOduration=6.060527614 podStartE2EDuration="7.119889916s" totalImagesPullingTime="998.391303ms" totalInitContainerRuntime="60.970999ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:18.078280076 +0000 UTC m=+159.916744028" watchObservedRunningTime="2026-09-18 19:59:22.119889916 +0000 UTC m=+163.958353912"
Sep 18 19:59:22.120261 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:22.120231    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9h4s5"]
Sep 18 19:59:22.125845 ip-10-0-132-172 kubenswrapper[2578]: W0918 19:59:22.125815    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9cb99c_d1fd_46f8_ae30_981161eaf053.slice/crio-8ff9511721c4eada24d35e0e25a913e027ed14ac577d287ccc836e8d79170020 WatchSource:0}: Error finding container 8ff9511721c4eada24d35e0e25a913e027ed14ac577d287ccc836e8d79170020: Status 404 returned error can't find the container with id 8ff9511721c4eada24d35e0e25a913e027ed14ac577d287ccc836e8d79170020
Sep 18 19:59:23.057458 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:23.057422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9h4s5" event={"ID":"6a9cb99c-d1fd-46f8-ae30-981161eaf053","Type":"ContainerStarted","Data":"8ff9511721c4eada24d35e0e25a913e027ed14ac577d287ccc836e8d79170020"}
Sep 18 19:59:23.586845 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:23.586803    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:23.733530 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:23.733480    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:24.062607 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:24.062576    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9h4s5" event={"ID":"6a9cb99c-d1fd-46f8-ae30-981161eaf053","Type":"ContainerStarted","Data":"8f06f4a91274eb6831966bac47525918b766d622162cc4d731f1036631c3783e"}
Sep 18 19:59:24.062996 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:24.062631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9h4s5" event={"ID":"6a9cb99c-d1fd-46f8-ae30-981161eaf053","Type":"ContainerStarted","Data":"1f66df0cfdfc665133140fd24f72a5e3c8e7b72ec5146d7c28933178d70e4478"}
Sep 18 19:59:24.062996 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:24.062685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:59:24.086701 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:24.086655    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9h4s5" podStartSLOduration=129.382740058 podStartE2EDuration="2m11.086640871s" totalImagesPullingTime="1.703900813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:24.085198043 +0000 UTC m=+165.923661994" watchObservedRunningTime="2026-09-18 19:59:24.086640871 +0000 UTC m=+165.925104825"
Sep 18 19:59:26.623050 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:26.623017    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:26.693967 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:26.693917    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:33.520191 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:33.520157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:33.651468 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:33.651428    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:34.069339 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:34.069310    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9h4s5"
Sep 18 19:59:43.235583 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:43.235544    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:59.438692 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:59.438660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:59.502416 ip-10-0-132-172 kubenswrapper[2578]: I0918 19:59:59.502379    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:14.693300 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:00:14.693269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:19.545527 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:00:19.545489    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:34.131061 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:00:34.131014    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:34.332207 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:00:34.332174    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:43.780770 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:00:43.780732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:43.854311 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:00:43.854273    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:54.976463 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:00:54.976374    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:55.067254 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:00:55.067218    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:15.461459 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:01:15.461423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:15.547045 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:01:15.547014    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:40.179637 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:01:40.179605    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 20:01:49.436235 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:01:49.436190    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:49.505609 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:01:49.505579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:54.688136 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:01:54.688099    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:58.623533 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:01:58.623499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:58.731141 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:01:58.731111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:01.391843 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:02:01.391802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:01.464234 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:02:01.464199    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:09.114118 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:02:09.114074    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:09.223053 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:02:09.223021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:11.619454 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:02:11.619408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:11.717529 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:02:11.717494    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:50.331067 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:02:50.331030    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:50.470951 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:02:50.470901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:02:50.571435 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:02:50.571401    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:12.211183 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:12.211146    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:13.494737 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:13.494698    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:16.281972 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:16.281913    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:16.379056 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:16.379024    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:21.559910 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:21.559869    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:21.698367 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:21.698335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:25.331925 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:25.331885    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:25.485497 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:25.485462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:25.637046 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:25.636970    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:25.760886 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:25.760849    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:27.375204 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.375169    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"]
Sep 18 20:03:27.377270 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.377254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"
Sep 18 20:03:27.379796 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.379775    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 20:03:27.379796 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.379774    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 20:03:27.380418 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.380401    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zjfq9\""
Sep 18 20:03:27.388235 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.388216    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"]
Sep 18 20:03:27.440203 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.440169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dadef34b-e498-4349-b7bd-29accd23268d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g\" (UID: \"dadef34b-e498-4349-b7bd-29accd23268d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"
Sep 18 20:03:27.440349 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.440215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dadef34b-e498-4349-b7bd-29accd23268d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g\" (UID: \"dadef34b-e498-4349-b7bd-29accd23268d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"
Sep 18 20:03:27.440349 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.440304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6krkl\" (UniqueName: \"kubernetes.io/projected/dadef34b-e498-4349-b7bd-29accd23268d-kube-api-access-6krkl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g\" (UID: \"dadef34b-e498-4349-b7bd-29accd23268d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"
Sep 18 20:03:27.505537 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.505503    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:27.541579 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.541552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dadef34b-e498-4349-b7bd-29accd23268d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g\" (UID: \"dadef34b-e498-4349-b7bd-29accd23268d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"
Sep 18 20:03:27.541695 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.541608    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krkl\" (UniqueName: \"kubernetes.io/projected/dadef34b-e498-4349-b7bd-29accd23268d-kube-api-access-6krkl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g\" (UID: \"dadef34b-e498-4349-b7bd-29accd23268d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"
Sep 18 20:03:27.541695 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.541647    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dadef34b-e498-4349-b7bd-29accd23268d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g\" (UID: \"dadef34b-e498-4349-b7bd-29accd23268d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"
Sep 18 20:03:27.542022 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.542003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dadef34b-e498-4349-b7bd-29accd23268d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g\" (UID: \"dadef34b-e498-4349-b7bd-29accd23268d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"
Sep 18 20:03:27.542022 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.542015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dadef34b-e498-4349-b7bd-29accd23268d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g\" (UID: \"dadef34b-e498-4349-b7bd-29accd23268d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"
Sep 18 20:03:27.550273 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.550251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krkl\" (UniqueName: \"kubernetes.io/projected/dadef34b-e498-4349-b7bd-29accd23268d-kube-api-access-6krkl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g\" (UID: \"dadef34b-e498-4349-b7bd-29accd23268d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"
Sep 18 20:03:27.686628 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.686547    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"
Sep 18 20:03:27.960415 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.960348    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"]
Sep 18 20:03:27.963559 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:03:27.963533    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadef34b_e498_4349_b7bd_29accd23268d.slice/crio-26b37cfa5b2d70d856ece2fbd07abb1be20b4b1db9788be1ac4df5f9dd0592f4 WatchSource:0}: Error finding container 26b37cfa5b2d70d856ece2fbd07abb1be20b4b1db9788be1ac4df5f9dd0592f4: Status 404 returned error can't find the container with id 26b37cfa5b2d70d856ece2fbd07abb1be20b4b1db9788be1ac4df5f9dd0592f4
Sep 18 20:03:27.965530 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:27.965511    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:03:28.115989 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:28.115961    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:28.849072 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:28.849031    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g" event={"ID":"dadef34b-e498-4349-b7bd-29accd23268d","Type":"ContainerStarted","Data":"26b37cfa5b2d70d856ece2fbd07abb1be20b4b1db9788be1ac4df5f9dd0592f4"}
Sep 18 20:03:34.872481 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:34.872444    2578 generic.go:356] "Generic (PLEG): container finished" podID="dadef34b-e498-4349-b7bd-29accd23268d" containerID="ba7200102549c5b8653487ea3e41f7028f75bbcb2c05494b9f02aa226674b030" exitCode=0
Sep 18 20:03:34.872873 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:34.872536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g" event={"ID":"dadef34b-e498-4349-b7bd-29accd23268d","Type":"ContainerDied","Data":"ba7200102549c5b8653487ea3e41f7028f75bbcb2c05494b9f02aa226674b030"}
Sep 18 20:03:35.932099 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:35.932057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:36.438500 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:36.438470    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-whknf"]
Sep 18 20:03:36.442961 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:36.442921    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-whknf"
Sep 18 20:03:36.445611 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:36.445587    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-zhh5n\""
Sep 18 20:03:36.445743 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:36.445588    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 20:03:36.446153 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:36.446129    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 20:03:36.455975 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:36.455952    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-whknf"]
Sep 18 20:03:36.510286 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:36.510255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8-bound-sa-token\") pod \"cert-manager-55b4d954d5-whknf\" (UID: \"3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8\") " pod="cert-manager/cert-manager-55b4d954d5-whknf"
Sep 18 20:03:36.510470 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:36.510402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f88sx\" (UniqueName: \"kubernetes.io/projected/3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8-kube-api-access-f88sx\") pod \"cert-manager-55b4d954d5-whknf\" (UID: \"3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8\") " pod="cert-manager/cert-manager-55b4d954d5-whknf"
Sep 18 20:03:36.611610 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:36.611574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f88sx\" (UniqueName: \"kubernetes.io/projected/3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8-kube-api-access-f88sx\") pod \"cert-manager-55b4d954d5-whknf\" (UID: \"3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8\") " pod="cert-manager/cert-manager-55b4d954d5-whknf"
Sep 18 20:03:36.611610 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:36.611616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8-bound-sa-token\") pod \"cert-manager-55b4d954d5-whknf\" (UID: \"3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8\") " pod="cert-manager/cert-manager-55b4d954d5-whknf"
Sep 18 20:03:36.621685 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:36.621641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8-bound-sa-token\") pod \"cert-manager-55b4d954d5-whknf\" (UID: \"3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8\") " pod="cert-manager/cert-manager-55b4d954d5-whknf"
Sep 18 20:03:36.623746 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:36.623723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f88sx\" (UniqueName: \"kubernetes.io/projected/3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8-kube-api-access-f88sx\") pod \"cert-manager-55b4d954d5-whknf\" (UID: \"3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8\") " pod="cert-manager/cert-manager-55b4d954d5-whknf"
Sep 18 20:03:36.754789 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:36.754760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-whknf"
Sep 18 20:03:37.271736 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:37.271707    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-whknf"]
Sep 18 20:03:37.274858 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:03:37.274831    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bb9efa2_cd88_4e3e_93ff_fd3bba2368b8.slice/crio-8fbd64de60c509f82f5a00ef93bf97b937312d2893a4d8e79fc8ee870fd29624 WatchSource:0}: Error finding container 8fbd64de60c509f82f5a00ef93bf97b937312d2893a4d8e79fc8ee870fd29624: Status 404 returned error can't find the container with id 8fbd64de60c509f82f5a00ef93bf97b937312d2893a4d8e79fc8ee870fd29624
Sep 18 20:03:37.887157 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:37.887069    2578 generic.go:356] "Generic (PLEG): container finished" podID="dadef34b-e498-4349-b7bd-29accd23268d" containerID="393cd3a6668eb5526d58cdb92cbdc52c996a5ee2f08da0b58e26c2efb2a6adef" exitCode=0
Sep 18 20:03:37.887322 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:37.887156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g" event={"ID":"dadef34b-e498-4349-b7bd-29accd23268d","Type":"ContainerDied","Data":"393cd3a6668eb5526d58cdb92cbdc52c996a5ee2f08da0b58e26c2efb2a6adef"}
Sep 18 20:03:37.888443 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:37.888419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-whknf" event={"ID":"3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8","Type":"ContainerStarted","Data":"8fbd64de60c509f82f5a00ef93bf97b937312d2893a4d8e79fc8ee870fd29624"}
Sep 18 20:03:40.905402 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:40.905347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-whknf" event={"ID":"3bb9efa2-cd88-4e3e-93ff-fd3bba2368b8","Type":"ContainerStarted","Data":"5c67c813ff9a7167d0c14a4aa89f58a55a06463bb34ee63c9b486a52714f20f2"}
Sep 18 20:03:40.923351 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:40.923299    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-whknf" podStartSLOduration=1.411762697 podStartE2EDuration="4.923283609s" totalImagesPullingTime="3.511520912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:40.922539364 +0000 UTC m=+422.761003317" watchObservedRunningTime="2026-09-18 20:03:40.923283609 +0000 UTC m=+422.761747560"
Sep 18 20:03:45.927250 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:45.927213    2578 generic.go:356] "Generic (PLEG): container finished" podID="dadef34b-e498-4349-b7bd-29accd23268d" containerID="3ddf751e9da2fe82539fd199d2e2d0ea47a99eab125f06284fe909a4db65594a" exitCode=0
Sep 18 20:03:45.927616 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:45.927295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g" event={"ID":"dadef34b-e498-4349-b7bd-29accd23268d","Type":"ContainerDied","Data":"3ddf751e9da2fe82539fd199d2e2d0ea47a99eab125f06284fe909a4db65594a"}
Sep 18 20:03:47.048504 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.048478    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"
Sep 18 20:03:47.196980 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.196869    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6krkl\" (UniqueName: \"kubernetes.io/projected/dadef34b-e498-4349-b7bd-29accd23268d-kube-api-access-6krkl\") pod \"dadef34b-e498-4349-b7bd-29accd23268d\" (UID: \"dadef34b-e498-4349-b7bd-29accd23268d\") "
Sep 18 20:03:47.196980 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.196908    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dadef34b-e498-4349-b7bd-29accd23268d-bundle\") pod \"dadef34b-e498-4349-b7bd-29accd23268d\" (UID: \"dadef34b-e498-4349-b7bd-29accd23268d\") "
Sep 18 20:03:47.196980 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.196964    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dadef34b-e498-4349-b7bd-29accd23268d-util\") pod \"dadef34b-e498-4349-b7bd-29accd23268d\" (UID: \"dadef34b-e498-4349-b7bd-29accd23268d\") "
Sep 18 20:03:47.197351 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.197325    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dadef34b-e498-4349-b7bd-29accd23268d-bundle" (OuterVolumeSpecName: "bundle") pod "dadef34b-e498-4349-b7bd-29accd23268d" (UID: "dadef34b-e498-4349-b7bd-29accd23268d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:47.199079 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.199055    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadef34b-e498-4349-b7bd-29accd23268d-kube-api-access-6krkl" (OuterVolumeSpecName: "kube-api-access-6krkl") pod "dadef34b-e498-4349-b7bd-29accd23268d" (UID: "dadef34b-e498-4349-b7bd-29accd23268d"). InnerVolumeSpecName "kube-api-access-6krkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:03:47.200971 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.200949    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dadef34b-e498-4349-b7bd-29accd23268d-util" (OuterVolumeSpecName: "util") pod "dadef34b-e498-4349-b7bd-29accd23268d" (UID: "dadef34b-e498-4349-b7bd-29accd23268d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:47.298021 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.297988    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6krkl\" (UniqueName: \"kubernetes.io/projected/dadef34b-e498-4349-b7bd-29accd23268d-kube-api-access-6krkl\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:03:47.298021 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.298015    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dadef34b-e498-4349-b7bd-29accd23268d-bundle\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:03:47.298021 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.298024    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dadef34b-e498-4349-b7bd-29accd23268d-util\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:03:47.859990 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.859956    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:47.935494 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.935466    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g" event={"ID":"dadef34b-e498-4349-b7bd-29accd23268d","Type":"ContainerDied","Data":"26b37cfa5b2d70d856ece2fbd07abb1be20b4b1db9788be1ac4df5f9dd0592f4"}
Sep 18 20:03:47.935645 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.935503    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxrl2g"
Sep 18 20:03:47.935645 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.935513    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26b37cfa5b2d70d856ece2fbd07abb1be20b4b1db9788be1ac4df5f9dd0592f4"
Sep 18 20:03:47.964538 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:47.964510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:48.542310 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:48.542271    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:48.761990 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:48.761951    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:48.852816 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:48.852733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:55.410865 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:55.410781    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:55.513890 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:55.513855    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:57.302005 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:03:57.301969    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:01.547181 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:01.547149    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:01.659224 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:01.659191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:05.230259 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:05.230218    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:05.333456 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:05.333420    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:05.431331 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:05.431290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:05.938853 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:05.938814    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:13.068768 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:13.068734    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:13.163482 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:13.163438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:13.518512 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:13.518470    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:13.914840 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:13.914751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:14.017536 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.017505    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:14.043295 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.043265    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"]
Sep 18 20:04:14.043496 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.043485    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dadef34b-e498-4349-b7bd-29accd23268d" containerName="extract"
Sep 18 20:04:14.043539 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.043498    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dadef34b-e498-4349-b7bd-29accd23268d" containerName="extract"
Sep 18 20:04:14.043539 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.043506    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dadef34b-e498-4349-b7bd-29accd23268d" containerName="util"
Sep 18 20:04:14.043539 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.043511    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dadef34b-e498-4349-b7bd-29accd23268d" containerName="util"
Sep 18 20:04:14.043539 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.043521    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dadef34b-e498-4349-b7bd-29accd23268d" containerName="pull"
Sep 18 20:04:14.043539 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.043526    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dadef34b-e498-4349-b7bd-29accd23268d" containerName="pull"
Sep 18 20:04:14.043684 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.043571    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dadef34b-e498-4349-b7bd-29accd23268d" containerName="extract"
Sep 18 20:04:14.117133 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.117100    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"]
Sep 18 20:04:14.117545 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.117199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.119927 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.119904    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 20:04:14.119927 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.119920    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 20:04:14.120418 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.120402    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 20:04:14.120463 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.120405    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cphdw\""
Sep 18 20:04:14.120677 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.120663    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 20:04:14.178145 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.178075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32db5bf8-5c32-4712-abef-7f1cf31b5561-tmp\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.178145 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.178105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32db5bf8-5c32-4712-abef-7f1cf31b5561-webhook-cert\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.178145 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.178138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvcf\" (UniqueName: \"kubernetes.io/projected/32db5bf8-5c32-4712-abef-7f1cf31b5561-kube-api-access-zvvcf\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.178341 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.178184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32db5bf8-5c32-4712-abef-7f1cf31b5561-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.178341 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.178227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32db5bf8-5c32-4712-abef-7f1cf31b5561-manifests\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.266200 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.266168    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:14.279209 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.279178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32db5bf8-5c32-4712-abef-7f1cf31b5561-manifests\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.279209 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.279210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32db5bf8-5c32-4712-abef-7f1cf31b5561-tmp\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.279399 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.279229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32db5bf8-5c32-4712-abef-7f1cf31b5561-webhook-cert\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.279399 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.279265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvcf\" (UniqueName: \"kubernetes.io/projected/32db5bf8-5c32-4712-abef-7f1cf31b5561-kube-api-access-zvvcf\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.279399 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.279306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32db5bf8-5c32-4712-abef-7f1cf31b5561-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.279569 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.279550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32db5bf8-5c32-4712-abef-7f1cf31b5561-manifests\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.281634 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.281609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32db5bf8-5c32-4712-abef-7f1cf31b5561-tmp\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.281736 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.281679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32db5bf8-5c32-4712-abef-7f1cf31b5561-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.281783 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.281767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32db5bf8-5c32-4712-abef-7f1cf31b5561-webhook-cert\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.291913 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.291881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvcf\" (UniqueName: \"kubernetes.io/projected/32db5bf8-5c32-4712-abef-7f1cf31b5561-kube-api-access-zvvcf\") pod \"opendatahub-operator-controller-manager-f76f79b89-cv5qp\" (UID: \"32db5bf8-5c32-4712-abef-7f1cf31b5561\") " pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.384989 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.384954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:14.426925 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.426890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:14.578677 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.578636    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:14.698469 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.698434    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:14.741965 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.741785    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"]
Sep 18 20:04:14.744666 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:04:14.744644    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32db5bf8_5c32_4712_abef_7f1cf31b5561.slice/crio-071ffb6eab244d75164c12295c3ce95804d8a09a77318e13adbf0463e2dfa824 WatchSource:0}: Error finding container 071ffb6eab244d75164c12295c3ce95804d8a09a77318e13adbf0463e2dfa824: Status 404 returned error can't find the container with id 071ffb6eab244d75164c12295c3ce95804d8a09a77318e13adbf0463e2dfa824
Sep 18 20:04:14.808399 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:14.808369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:15.025531 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:15.025457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp" event={"ID":"32db5bf8-5c32-4712-abef-7f1cf31b5561","Type":"ContainerStarted","Data":"071ffb6eab244d75164c12295c3ce95804d8a09a77318e13adbf0463e2dfa824"}
Sep 18 20:04:15.951095 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:15.951059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:16.055294 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:16.055261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:16.150304 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:16.150260    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:19.040522 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:19.040481    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp" event={"ID":"32db5bf8-5c32-4712-abef-7f1cf31b5561","Type":"ContainerStarted","Data":"7ac895018b11a7186bc3d3fbb740bff8cea432b374b0a71dced5773ff6f94e52"}
Sep 18 20:04:20.046663 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:20.046630    2578 generic.go:356] "Generic (PLEG): container finished" podID="32db5bf8-5c32-4712-abef-7f1cf31b5561" containerID="7ac895018b11a7186bc3d3fbb740bff8cea432b374b0a71dced5773ff6f94e52" exitCode=0
Sep 18 20:04:20.047068 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:20.046676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp" event={"ID":"32db5bf8-5c32-4712-abef-7f1cf31b5561","Type":"ContainerDied","Data":"7ac895018b11a7186bc3d3fbb740bff8cea432b374b0a71dced5773ff6f94e52"}
Sep 18 20:04:21.051434 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:21.051391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp" event={"ID":"32db5bf8-5c32-4712-abef-7f1cf31b5561","Type":"ContainerStarted","Data":"f106d19ec6a57d3e2b40fc5bff657b89eaeed1dae5f99cb53194548926870c18"}
Sep 18 20:04:21.051899 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:21.051510    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:21.077032 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:21.076992    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp" podStartSLOduration=2.13666516 podStartE2EDuration="7.076977491s" totalImagesPullingTime="3.859762338s" totalInitContainerRuntime="1.080549993s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:21.07511391 +0000 UTC m=+462.913577862" watchObservedRunningTime="2026-09-18 20:04:21.076977491 +0000 UTC m=+462.915441443"
Sep 18 20:04:32.057656 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:32.057626    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f76f79b89-cv5qp"
Sep 18 20:04:33.926401 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:33.926365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:34.048791 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.048756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:34.390641 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.390608    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:34.545496 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.545464    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"]
Sep 18 20:04:34.550102 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.550083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"
Sep 18 20:04:34.567784 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.567760    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 20:04:34.568263 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.568240    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 20:04:34.568379 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.568243    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 20:04:34.568379 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.568310    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 20:04:34.591803 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.591779    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"]
Sep 18 20:04:34.612044 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.612016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66fe211b-0f00-4ee4-b678-9224e74a144c-tls-certs\") pod \"kube-auth-proxy-6dc98cf6f8-p6m5k\" (UID: \"66fe211b-0f00-4ee4-b678-9224e74a144c\") " pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"
Sep 18 20:04:34.612161 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.612045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66fe211b-0f00-4ee4-b678-9224e74a144c-tmp\") pod \"kube-auth-proxy-6dc98cf6f8-p6m5k\" (UID: \"66fe211b-0f00-4ee4-b678-9224e74a144c\") " pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"
Sep 18 20:04:34.612161 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.612067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r76xn\" (UniqueName: \"kubernetes.io/projected/66fe211b-0f00-4ee4-b678-9224e74a144c-kube-api-access-r76xn\") pod \"kube-auth-proxy-6dc98cf6f8-p6m5k\" (UID: \"66fe211b-0f00-4ee4-b678-9224e74a144c\") " pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"
Sep 18 20:04:34.713216 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.713190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66fe211b-0f00-4ee4-b678-9224e74a144c-tls-certs\") pod \"kube-auth-proxy-6dc98cf6f8-p6m5k\" (UID: \"66fe211b-0f00-4ee4-b678-9224e74a144c\") " pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"
Sep 18 20:04:34.713386 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.713219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66fe211b-0f00-4ee4-b678-9224e74a144c-tmp\") pod \"kube-auth-proxy-6dc98cf6f8-p6m5k\" (UID: \"66fe211b-0f00-4ee4-b678-9224e74a144c\") " pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"
Sep 18 20:04:34.713386 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.713239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r76xn\" (UniqueName: \"kubernetes.io/projected/66fe211b-0f00-4ee4-b678-9224e74a144c-kube-api-access-r76xn\") pod \"kube-auth-proxy-6dc98cf6f8-p6m5k\" (UID: \"66fe211b-0f00-4ee4-b678-9224e74a144c\") " pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"
Sep 18 20:04:34.715571 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.715541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66fe211b-0f00-4ee4-b678-9224e74a144c-tmp\") pod \"kube-auth-proxy-6dc98cf6f8-p6m5k\" (UID: \"66fe211b-0f00-4ee4-b678-9224e74a144c\") " pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"
Sep 18 20:04:34.715775 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.715755    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66fe211b-0f00-4ee4-b678-9224e74a144c-tls-certs\") pod \"kube-auth-proxy-6dc98cf6f8-p6m5k\" (UID: \"66fe211b-0f00-4ee4-b678-9224e74a144c\") " pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"
Sep 18 20:04:34.820246 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.820209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r76xn\" (UniqueName: \"kubernetes.io/projected/66fe211b-0f00-4ee4-b678-9224e74a144c-kube-api-access-r76xn\") pod \"kube-auth-proxy-6dc98cf6f8-p6m5k\" (UID: \"66fe211b-0f00-4ee4-b678-9224e74a144c\") " pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"
Sep 18 20:04:34.858983 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.858919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"
Sep 18 20:04:34.863458 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:34.863426    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:35.177540 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:35.177425    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k"]
Sep 18 20:04:35.181113 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:04:35.181083    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fe211b_0f00_4ee4_b678_9224e74a144c.slice/crio-5a73e76b76fcb44bd7765addfcd322a66c49297a9acd858398c1284bb2a6c46d WatchSource:0}: Error finding container 5a73e76b76fcb44bd7765addfcd322a66c49297a9acd858398c1284bb2a6c46d: Status 404 returned error can't find the container with id 5a73e76b76fcb44bd7765addfcd322a66c49297a9acd858398c1284bb2a6c46d
Sep 18 20:04:36.107098 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:36.107054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k" event={"ID":"66fe211b-0f00-4ee4-b678-9224e74a144c","Type":"ContainerStarted","Data":"5a73e76b76fcb44bd7765addfcd322a66c49297a9acd858398c1284bb2a6c46d"}
Sep 18 20:04:37.659125 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.659059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:37.697206 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.697174    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"]
Sep 18 20:04:37.699507 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.699483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.705384 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.705354    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 20:04:37.705384 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.705356    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xb9s2\""
Sep 18 20:04:37.705715 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.705689    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 20:04:37.705828 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.705737    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 20:04:37.706859 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.706838    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 20:04:37.727824 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.727794    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"]
Sep 18 20:04:37.739071 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.739042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.739213 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.739101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.739213 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.739168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d9vk\" (UniqueName: \"kubernetes.io/projected/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-kube-api-access-8d9vk\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.739326 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.739216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.739326 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.739246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.739326 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.739319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.739466 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.739353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.819197 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.819151    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:37.840185 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.840156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.840185 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.840189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d9vk\" (UniqueName: \"kubernetes.io/projected/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-kube-api-access-8d9vk\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.840418 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.840213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.840418 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.840385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.840519 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.840486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.840569 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.840521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.840569 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.840550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.841058 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.841029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.843122 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.843079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.843292 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.843268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.843416 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.843393    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.843505 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.843446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.852461 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.852440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d9vk\" (UniqueName: \"kubernetes.io/projected/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-kube-api-access-8d9vk\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:37.852567 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:37.852528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vp9n5\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:38.010745 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:38.010724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:38.116034 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:38.115996    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k" event={"ID":"66fe211b-0f00-4ee4-b678-9224e74a144c","Type":"ContainerStarted","Data":"4e636076faf06e28cef67a033be22cd1643c7828a2889610bdf74ec39e62716b"}
Sep 18 20:04:38.136408 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:38.136355    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6dc98cf6f8-p6m5k" podStartSLOduration=1.358880557 podStartE2EDuration="4.136336403s" totalImagesPullingTime="2.777455846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:38.135062274 +0000 UTC m=+479.973526226" watchObservedRunningTime="2026-09-18 20:04:38.136336403 +0000 UTC m=+479.974800358"
Sep 18 20:04:38.290431 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:38.290400    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"]
Sep 18 20:04:38.293279 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:04:38.293244    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49633e0_0bd3_4c27_9eaa_9d6c15c482ba.slice/crio-7ac8c5e05bbb1579961ce04917946db069001e5876665560519cdd3e69054f73 WatchSource:0}: Error finding container 7ac8c5e05bbb1579961ce04917946db069001e5876665560519cdd3e69054f73: Status 404 returned error can't find the container with id 7ac8c5e05bbb1579961ce04917946db069001e5876665560519cdd3e69054f73
Sep 18 20:04:39.028320 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:39.028262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:39.122415 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:39.122365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5" event={"ID":"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba","Type":"ContainerStarted","Data":"7ac8c5e05bbb1579961ce04917946db069001e5876665560519cdd3e69054f73"}
Sep 18 20:04:40.654278 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:40.654228    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 20:04:40.654569 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:40.654317    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 20:04:41.131791 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:41.131746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5" event={"ID":"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba","Type":"ContainerStarted","Data":"ad32a742b856c35ff913273e5ee586f796e77eb95be17433340b51ff2d0fbce9"}
Sep 18 20:04:41.132028 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:41.131940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:41.133450 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:41.133424    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vp9n5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 20:04:41.133555 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:41.133470    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5" podUID="f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:04:41.153299 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:41.153250    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5" podStartSLOduration=1.794532139 podStartE2EDuration="4.153234979s" totalImagesPullingTime="2.35870284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:41.152447663 +0000 UTC m=+482.990911619" watchObservedRunningTime="2026-09-18 20:04:41.153234979 +0000 UTC m=+482.991698933"
Sep 18 20:04:41.378454 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:41.378418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:41.465620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:41.465584    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:42.137070 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:42.137038    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vp9n5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 20:04:42.137426 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:42.137091    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5" podUID="f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:04:42.176065 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:42.176025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:42.352195 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:42.352165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:42.498433 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:42.498387    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:45.138047 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:45.138014    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:45.279598 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:45.279558    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:45.435493 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:45.435412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:45.577267 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:45.577225    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:45.895876 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:45.895846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:46.099451 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:46.099417    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:53.451771 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.451734    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"]
Sep 18 20:04:53.454775 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.454759    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.475012 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.474991    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"]
Sep 18 20:04:53.564642 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.564605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807e886a-5281-4441-9363-a16a16855842-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.564642 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.564644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/807e886a-5281-4441-9363-a16a16855842-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.564893 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.564667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/807e886a-5281-4441-9363-a16a16855842-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.564893 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.564779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/807e886a-5281-4441-9363-a16a16855842-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.564893 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.564815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvwq\" (UniqueName: \"kubernetes.io/projected/807e886a-5281-4441-9363-a16a16855842-kube-api-access-kzvwq\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.564893 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.564859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/807e886a-5281-4441-9363-a16a16855842-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.565069 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.564952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/807e886a-5281-4441-9363-a16a16855842-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.666260 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.666224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/807e886a-5281-4441-9363-a16a16855842-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.666428 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.666281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/807e886a-5281-4441-9363-a16a16855842-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.666428 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.666376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807e886a-5281-4441-9363-a16a16855842-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.666428 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.666393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/807e886a-5281-4441-9363-a16a16855842-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.666428 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.666413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/807e886a-5281-4441-9363-a16a16855842-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.666700 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.666434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/807e886a-5281-4441-9363-a16a16855842-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.666700 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.666449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvwq\" (UniqueName: \"kubernetes.io/projected/807e886a-5281-4441-9363-a16a16855842-kube-api-access-kzvwq\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.667171 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.667143    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/807e886a-5281-4441-9363-a16a16855842-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.668767 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.668739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/807e886a-5281-4441-9363-a16a16855842-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.668862 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.668769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/807e886a-5281-4441-9363-a16a16855842-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.668862 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.668815    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/807e886a-5281-4441-9363-a16a16855842-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.668993 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.668969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/807e886a-5281-4441-9363-a16a16855842-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.676635 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.676613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807e886a-5281-4441-9363-a16a16855842-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.678069 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.678047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvwq\" (UniqueName: \"kubernetes.io/projected/807e886a-5281-4441-9363-a16a16855842-kube-api-access-kzvwq\") pod \"istiod-openshift-gateway-5677ccb49b-j8kd5\" (UID: \"807e886a-5281-4441-9363-a16a16855842\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:53.763499 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:53.763460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:54.061861 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:54.061786    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"]
Sep 18 20:04:54.064877 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:04:54.064847    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807e886a_5281_4441_9363_a16a16855842.slice/crio-8de799721e9a602f0aafa02360a726bbde2e4e0fb230b855078b99c8d994960c WatchSource:0}: Error finding container 8de799721e9a602f0aafa02360a726bbde2e4e0fb230b855078b99c8d994960c: Status 404 returned error can't find the container with id 8de799721e9a602f0aafa02360a726bbde2e4e0fb230b855078b99c8d994960c
Sep 18 20:04:54.067031 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:54.066999    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 20:04:54.067124 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:54.067073    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 20:04:54.178048 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:54.178008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5" event={"ID":"807e886a-5281-4441-9363-a16a16855842","Type":"ContainerStarted","Data":"54b01f4df07f379c2f75f64e83a53f8ea66e40d0fbd3d140ea8a85e4071eaf4c"}
Sep 18 20:04:54.178228 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:54.178064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5" event={"ID":"807e886a-5281-4441-9363-a16a16855842","Type":"ContainerStarted","Data":"8de799721e9a602f0aafa02360a726bbde2e4e0fb230b855078b99c8d994960c"}
Sep 18 20:04:54.178228 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:54.178115    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:54.204475 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:54.204424    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5" podStartSLOduration=1.204408196 podStartE2EDuration="1.204408196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:54.2032038 +0000 UTC m=+496.041667755" watchObservedRunningTime="2026-09-18 20:04:54.204408196 +0000 UTC m=+496.042872148"
Sep 18 20:04:55.183895 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.183868    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j8kd5"
Sep 18 20:04:55.327099 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.327064    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"]
Sep 18 20:04:55.327256 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.327236    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5" podUID="f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" containerName="discovery" containerID="cri-o://ad32a742b856c35ff913273e5ee586f796e77eb95be17433340b51ff2d0fbce9" gracePeriod=30
Sep 18 20:04:55.561788 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.561756    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:55.684637 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.684590    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-cacerts\") pod \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") "
Sep 18 20:04:55.684804 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.684644    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-csr-dns-cert\") pod \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") "
Sep 18 20:04:55.684804 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.684740    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-token\") pod \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") "
Sep 18 20:04:55.684804 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.684774    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-kubeconfig\") pod \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") "
Sep 18 20:04:55.684929 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.684817    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-local-certs\") pod \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") "
Sep 18 20:04:55.684929 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.684842    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-csr-ca-configmap\") pod \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") "
Sep 18 20:04:55.684929 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.684883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d9vk\" (UniqueName: \"kubernetes.io/projected/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-kube-api-access-8d9vk\") pod \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\" (UID: \"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba\") "
Sep 18 20:04:55.685395 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.685358    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" (UID: "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:55.687168 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.687143    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" (UID: "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:55.687322 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.687292    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-cacerts" (OuterVolumeSpecName: "cacerts") pod "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" (UID: "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:55.687418 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.687319    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-kube-api-access-8d9vk" (OuterVolumeSpecName: "kube-api-access-8d9vk") pod "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" (UID: "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba"). InnerVolumeSpecName "kube-api-access-8d9vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:55.687418 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.687330    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" (UID: "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:55.687418 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.687394    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-local-certs" (OuterVolumeSpecName: "local-certs") pod "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" (UID: "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:55.687520 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.687453    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-token" (OuterVolumeSpecName: "istio-token") pod "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" (UID: "f49633e0-0bd3-4c27-9eaa-9d6c15c482ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:55.785750 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.785716    2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-cacerts\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:04:55.785750 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.785746    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-csr-dns-cert\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:04:55.785750 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.785758    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-token\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:04:55.785983 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.785766    2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-kubeconfig\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:04:55.785983 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.785774    2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-local-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:04:55.785983 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.785782    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-istio-csr-ca-configmap\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:04:55.785983 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:55.785791    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d9vk\" (UniqueName: \"kubernetes.io/projected/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba-kube-api-access-8d9vk\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:04:56.186999 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.186905    2578 generic.go:356] "Generic (PLEG): container finished" podID="f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" containerID="ad32a742b856c35ff913273e5ee586f796e77eb95be17433340b51ff2d0fbce9" exitCode=0
Sep 18 20:04:56.186999 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.186958    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"
Sep 18 20:04:56.186999 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.186991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5" event={"ID":"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba","Type":"ContainerDied","Data":"ad32a742b856c35ff913273e5ee586f796e77eb95be17433340b51ff2d0fbce9"}
Sep 18 20:04:56.187506 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.187039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5" event={"ID":"f49633e0-0bd3-4c27-9eaa-9d6c15c482ba","Type":"ContainerDied","Data":"7ac8c5e05bbb1579961ce04917946db069001e5876665560519cdd3e69054f73"}
Sep 18 20:04:56.187506 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.187073    2578 scope.go:117] "RemoveContainer" containerID="ad32a742b856c35ff913273e5ee586f796e77eb95be17433340b51ff2d0fbce9"
Sep 18 20:04:56.196241 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.196225    2578 scope.go:117] "RemoveContainer" containerID="ad32a742b856c35ff913273e5ee586f796e77eb95be17433340b51ff2d0fbce9"
Sep 18 20:04:56.196516 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:04:56.196498    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad32a742b856c35ff913273e5ee586f796e77eb95be17433340b51ff2d0fbce9\": container with ID starting with ad32a742b856c35ff913273e5ee586f796e77eb95be17433340b51ff2d0fbce9 not found: ID does not exist" containerID="ad32a742b856c35ff913273e5ee586f796e77eb95be17433340b51ff2d0fbce9"
Sep 18 20:04:56.196565 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.196523    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad32a742b856c35ff913273e5ee586f796e77eb95be17433340b51ff2d0fbce9"} err="failed to get container status \"ad32a742b856c35ff913273e5ee586f796e77eb95be17433340b51ff2d0fbce9\": rpc error: code = NotFound desc = could not find container \"ad32a742b856c35ff913273e5ee586f796e77eb95be17433340b51ff2d0fbce9\": container with ID starting with ad32a742b856c35ff913273e5ee586f796e77eb95be17433340b51ff2d0fbce9 not found: ID does not exist"
Sep 18 20:04:56.244828 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.244792    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"]
Sep 18 20:04:56.257542 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.257520    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vp9n5"]
Sep 18 20:04:56.321178 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.321146    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" path="/var/lib/kubelet/pods/f49633e0-0bd3-4c27-9eaa-9d6c15c482ba/volumes"
Sep 18 20:04:56.778469 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.778439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:56.877973 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.877923    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"]
Sep 18 20:04:56.878281 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.878264    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" containerName="discovery"
Sep 18 20:04:56.878358 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.878283    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" containerName="discovery"
Sep 18 20:04:56.878358 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.878341    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f49633e0-0bd3-4c27-9eaa-9d6c15c482ba" containerName="discovery"
Sep 18 20:04:56.882517 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.882494    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"
Sep 18 20:04:56.885665 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.885642    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 18 20:04:56.886100 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.886081    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-tbvh8\""
Sep 18 20:04:56.895366 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.895336    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"]
Sep 18 20:04:56.953411 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.953372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:56.996778 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.996742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcf7a688-09a8-4a24-8931-6692efa454b6-cert\") pod \"kserve-controller-manager-d6f4b65b8-wr6h9\" (UID: \"dcf7a688-09a8-4a24-8931-6692efa454b6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"
Sep 18 20:04:56.996778 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:56.996778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntm6r\" (UniqueName: \"kubernetes.io/projected/dcf7a688-09a8-4a24-8931-6692efa454b6-kube-api-access-ntm6r\") pod \"kserve-controller-manager-d6f4b65b8-wr6h9\" (UID: \"dcf7a688-09a8-4a24-8931-6692efa454b6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"
Sep 18 20:04:57.097578 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:57.097496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcf7a688-09a8-4a24-8931-6692efa454b6-cert\") pod \"kserve-controller-manager-d6f4b65b8-wr6h9\" (UID: \"dcf7a688-09a8-4a24-8931-6692efa454b6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"
Sep 18 20:04:57.097578 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:57.097531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntm6r\" (UniqueName: \"kubernetes.io/projected/dcf7a688-09a8-4a24-8931-6692efa454b6-kube-api-access-ntm6r\") pod \"kserve-controller-manager-d6f4b65b8-wr6h9\" (UID: \"dcf7a688-09a8-4a24-8931-6692efa454b6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"
Sep 18 20:04:57.097737 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:04:57.097640    2578 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 18 20:04:57.097737 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:04:57.097705    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf7a688-09a8-4a24-8931-6692efa454b6-cert podName:dcf7a688-09a8-4a24-8931-6692efa454b6 nodeName:}" failed. No retries permitted until 2026-09-18 20:04:57.597689683 +0000 UTC m=+499.436153613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/dcf7a688-09a8-4a24-8931-6692efa454b6-cert") pod "kserve-controller-manager-d6f4b65b8-wr6h9" (UID: "dcf7a688-09a8-4a24-8931-6692efa454b6") : secret "kserve-webhook-server-cert" not found
Sep 18 20:04:57.112219 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:57.112190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntm6r\" (UniqueName: \"kubernetes.io/projected/dcf7a688-09a8-4a24-8931-6692efa454b6-kube-api-access-ntm6r\") pod \"kserve-controller-manager-d6f4b65b8-wr6h9\" (UID: \"dcf7a688-09a8-4a24-8931-6692efa454b6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"
Sep 18 20:04:57.152345 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:57.152317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:57.310389 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:57.310356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:57.417524 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:57.417439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:57.602162 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:57.602128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcf7a688-09a8-4a24-8931-6692efa454b6-cert\") pod \"kserve-controller-manager-d6f4b65b8-wr6h9\" (UID: \"dcf7a688-09a8-4a24-8931-6692efa454b6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"
Sep 18 20:04:57.604488 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:57.604465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcf7a688-09a8-4a24-8931-6692efa454b6-cert\") pod \"kserve-controller-manager-d6f4b65b8-wr6h9\" (UID: \"dcf7a688-09a8-4a24-8931-6692efa454b6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"
Sep 18 20:04:57.792333 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:57.792298    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"
Sep 18 20:04:58.065182 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:58.065108    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"]
Sep 18 20:04:58.069278 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:04:58.069249    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf7a688_09a8_4a24_8931_6692efa454b6.slice/crio-26cc3bc5267a60365e3c655b59b0fb0f640a01635bbdfd3bfd5c2fc9c8f68a38 WatchSource:0}: Error finding container 26cc3bc5267a60365e3c655b59b0fb0f640a01635bbdfd3bfd5c2fc9c8f68a38: Status 404 returned error can't find the container with id 26cc3bc5267a60365e3c655b59b0fb0f640a01635bbdfd3bfd5c2fc9c8f68a38
Sep 18 20:04:58.196764 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:04:58.196733    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9" event={"ID":"dcf7a688-09a8-4a24-8931-6692efa454b6","Type":"ContainerStarted","Data":"26cc3bc5267a60365e3c655b59b0fb0f640a01635bbdfd3bfd5c2fc9c8f68a38"}
Sep 18 20:05:00.207032 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:00.206985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9" event={"ID":"dcf7a688-09a8-4a24-8931-6692efa454b6","Type":"ContainerStarted","Data":"38bb13b4e01a959c0505e1be6d7fcdbc1324faa64fde25fa3d16255080f26b76"}
Sep 18 20:05:00.207479 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:00.207077    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"
Sep 18 20:05:00.256111 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:00.256053    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9" podStartSLOduration=2.199681937 podStartE2EDuration="4.256033832s" totalImagesPullingTime="2.056351895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:00.254091284 +0000 UTC m=+502.092555230" watchObservedRunningTime="2026-09-18 20:05:00.256033832 +0000 UTC m=+502.094497783"
Sep 18 20:05:12.389385 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:12.389351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:12.493007 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:12.492968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:31.216616 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:31.216579    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wr6h9"
Sep 18 20:05:40.254203 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:40.254167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:40.352241 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:40.352201    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:45.959678 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:45.959646    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:46.052500 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:46.052466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:51.987695 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:51.987663    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"]
Sep 18 20:05:51.992596 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:51.992577    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"
Sep 18 20:05:51.996334 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:51.996315    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 20:05:51.996455 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:51.996313    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 20:05:51.997148 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:51.997129    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zjfq9\""
Sep 18 20:05:52.009271 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.009251    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"]
Sep 18 20:05:52.094601 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.094570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pqvf\" (UniqueName: \"kubernetes.io/projected/8c76ea29-391d-4b06-a762-66c2cf0b2101-kube-api-access-8pqvf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8\" (UID: \"8c76ea29-391d-4b06-a762-66c2cf0b2101\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"
Sep 18 20:05:52.094749 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.094623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c76ea29-391d-4b06-a762-66c2cf0b2101-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8\" (UID: \"8c76ea29-391d-4b06-a762-66c2cf0b2101\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"
Sep 18 20:05:52.094749 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.094648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c76ea29-391d-4b06-a762-66c2cf0b2101-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8\" (UID: \"8c76ea29-391d-4b06-a762-66c2cf0b2101\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"
Sep 18 20:05:52.097633 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.097605    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"]
Sep 18 20:05:52.100744 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.100728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"
Sep 18 20:05:52.114752 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.114729    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"]
Sep 18 20:05:52.195192 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.195162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pqvf\" (UniqueName: \"kubernetes.io/projected/8c76ea29-391d-4b06-a762-66c2cf0b2101-kube-api-access-8pqvf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8\" (UID: \"8c76ea29-391d-4b06-a762-66c2cf0b2101\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"
Sep 18 20:05:52.195380 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.195203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c76ea29-391d-4b06-a762-66c2cf0b2101-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8\" (UID: \"8c76ea29-391d-4b06-a762-66c2cf0b2101\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"
Sep 18 20:05:52.195380 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.195225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c76ea29-391d-4b06-a762-66c2cf0b2101-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8\" (UID: \"8c76ea29-391d-4b06-a762-66c2cf0b2101\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"
Sep 18 20:05:52.195380 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.195247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd\" (UID: \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"
Sep 18 20:05:52.195380 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.195268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hvxg\" (UniqueName: \"kubernetes.io/projected/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-kube-api-access-9hvxg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd\" (UID: \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"
Sep 18 20:05:52.195380 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.195326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd\" (UID: \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"
Sep 18 20:05:52.195656 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.195638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c76ea29-391d-4b06-a762-66c2cf0b2101-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8\" (UID: \"8c76ea29-391d-4b06-a762-66c2cf0b2101\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"
Sep 18 20:05:52.195709 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.195658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c76ea29-391d-4b06-a762-66c2cf0b2101-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8\" (UID: \"8c76ea29-391d-4b06-a762-66c2cf0b2101\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"
Sep 18 20:05:52.212363 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.212339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pqvf\" (UniqueName: \"kubernetes.io/projected/8c76ea29-391d-4b06-a762-66c2cf0b2101-kube-api-access-8pqvf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8\" (UID: \"8c76ea29-391d-4b06-a762-66c2cf0b2101\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"
Sep 18 20:05:52.273962 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.273873    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"]
Sep 18 20:05:52.277253 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.277238    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"
Sep 18 20:05:52.292128 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.292106    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"]
Sep 18 20:05:52.296636 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.296612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47\" (UID: \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"
Sep 18 20:05:52.296741 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.296656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd\" (UID: \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"
Sep 18 20:05:52.296741 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.296675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47\" (UID: \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"
Sep 18 20:05:52.296741 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.296700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pdzx\" (UniqueName: \"kubernetes.io/projected/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-kube-api-access-9pdzx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47\" (UID: \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"
Sep 18 20:05:52.296741 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.296723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hvxg\" (UniqueName: \"kubernetes.io/projected/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-kube-api-access-9hvxg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd\" (UID: \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"
Sep 18 20:05:52.296979 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.296750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd\" (UID: \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"
Sep 18 20:05:52.297043 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.296998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd\" (UID: \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"
Sep 18 20:05:52.297099 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.297083    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd\" (UID: \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"
Sep 18 20:05:52.301612 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.301596    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"
Sep 18 20:05:52.309842 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.309824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hvxg\" (UniqueName: \"kubernetes.io/projected/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-kube-api-access-9hvxg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd\" (UID: \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"
Sep 18 20:05:52.397689 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.397654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47\" (UID: \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"
Sep 18 20:05:52.397689 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.397692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdzx\" (UniqueName: \"kubernetes.io/projected/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-kube-api-access-9pdzx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47\" (UID: \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"
Sep 18 20:05:52.397864 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.397737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47\" (UID: \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"
Sep 18 20:05:52.398081 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.398061    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47\" (UID: \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"
Sep 18 20:05:52.398125 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.398067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47\" (UID: \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"
Sep 18 20:05:52.407867 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.407842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pdzx\" (UniqueName: \"kubernetes.io/projected/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-kube-api-access-9pdzx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47\" (UID: \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"
Sep 18 20:05:52.409592 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.409579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"
Sep 18 20:05:52.586638 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.586610    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"]
Sep 18 20:05:52.586734 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.586627    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"
Sep 18 20:05:52.589187 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:05:52.589164    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c76ea29_391d_4b06_a762_66c2cf0b2101.slice/crio-92fd0b8a8c96eb50acca9a76fb7009605c7aabfae6d011a11f60304e258f9d58 WatchSource:0}: Error finding container 92fd0b8a8c96eb50acca9a76fb7009605c7aabfae6d011a11f60304e258f9d58: Status 404 returned error can't find the container with id 92fd0b8a8c96eb50acca9a76fb7009605c7aabfae6d011a11f60304e258f9d58
Sep 18 20:05:52.686254 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.686230    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"]
Sep 18 20:05:52.689296 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:05:52.689269    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7c0dcb_8057_46f2_9768_2ef68036cbf4.slice/crio-9cc0d57f5a8735deb1069452de67440b57ca83120147a03026b439bdc2c05e55 WatchSource:0}: Error finding container 9cc0d57f5a8735deb1069452de67440b57ca83120147a03026b439bdc2c05e55: Status 404 returned error can't find the container with id 9cc0d57f5a8735deb1069452de67440b57ca83120147a03026b439bdc2c05e55
Sep 18 20:05:52.864568 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:52.864491    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"]
Sep 18 20:05:52.886778 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:05:52.886748    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96ae9aa_3fab_4ec8_a0be_d501e1fc3bf1.slice/crio-5f762744e62a393ebf53da27da15d81e8b859c7dd6622a21a8f958f3cbc63309 WatchSource:0}: Error finding container 5f762744e62a393ebf53da27da15d81e8b859c7dd6622a21a8f958f3cbc63309: Status 404 returned error can't find the container with id 5f762744e62a393ebf53da27da15d81e8b859c7dd6622a21a8f958f3cbc63309
Sep 18 20:05:53.413287 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:53.413256    2578 generic.go:356] "Generic (PLEG): container finished" podID="cc7c0dcb-8057-46f2-9768-2ef68036cbf4" containerID="dca45b325f9d70a1b376cd96ce7e1073dfe15d4f43a4c7f05ba5090d6a899786" exitCode=0
Sep 18 20:05:53.413714 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:53.413340    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd" event={"ID":"cc7c0dcb-8057-46f2-9768-2ef68036cbf4","Type":"ContainerDied","Data":"dca45b325f9d70a1b376cd96ce7e1073dfe15d4f43a4c7f05ba5090d6a899786"}
Sep 18 20:05:53.413714 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:53.413392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd" event={"ID":"cc7c0dcb-8057-46f2-9768-2ef68036cbf4","Type":"ContainerStarted","Data":"9cc0d57f5a8735deb1069452de67440b57ca83120147a03026b439bdc2c05e55"}
Sep 18 20:05:53.415077 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:53.415054    2578 generic.go:356] "Generic (PLEG): container finished" podID="e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1" containerID="b6ed09f236ea81ec1fbc451ccf5f9ae2556d3037055dc005f96fe4824012c1e9" exitCode=0
Sep 18 20:05:53.415165 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:53.415125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47" event={"ID":"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1","Type":"ContainerDied","Data":"b6ed09f236ea81ec1fbc451ccf5f9ae2556d3037055dc005f96fe4824012c1e9"}
Sep 18 20:05:53.415221 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:53.415163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47" event={"ID":"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1","Type":"ContainerStarted","Data":"5f762744e62a393ebf53da27da15d81e8b859c7dd6622a21a8f958f3cbc63309"}
Sep 18 20:05:53.416661 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:53.416636    2578 generic.go:356] "Generic (PLEG): container finished" podID="8c76ea29-391d-4b06-a762-66c2cf0b2101" containerID="5a3a530877a476c9dfbebb2bcc79ba65cb66913b304fe0b9e8484a4e371f8240" exitCode=0
Sep 18 20:05:53.416762 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:53.416660    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8" event={"ID":"8c76ea29-391d-4b06-a762-66c2cf0b2101","Type":"ContainerDied","Data":"5a3a530877a476c9dfbebb2bcc79ba65cb66913b304fe0b9e8484a4e371f8240"}
Sep 18 20:05:53.416762 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:53.416702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8" event={"ID":"8c76ea29-391d-4b06-a762-66c2cf0b2101","Type":"ContainerStarted","Data":"92fd0b8a8c96eb50acca9a76fb7009605c7aabfae6d011a11f60304e258f9d58"}
Sep 18 20:05:54.422741 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:54.422646    2578 generic.go:356] "Generic (PLEG): container finished" podID="cc7c0dcb-8057-46f2-9768-2ef68036cbf4" containerID="e15af1bc027fc0690a23fda66a153c8739d82ff0c4d5555e53b56bca38b8e444" exitCode=0
Sep 18 20:05:54.423214 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:54.422737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd" event={"ID":"cc7c0dcb-8057-46f2-9768-2ef68036cbf4","Type":"ContainerDied","Data":"e15af1bc027fc0690a23fda66a153c8739d82ff0c4d5555e53b56bca38b8e444"}
Sep 18 20:05:54.424583 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:54.424565    2578 generic.go:356] "Generic (PLEG): container finished" podID="e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1" containerID="8b1a608d7fce9c091fb9120d4e301565034a75ca4502be800fb3ea13b20d0e06" exitCode=0
Sep 18 20:05:54.424681 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:54.424637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47" event={"ID":"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1","Type":"ContainerDied","Data":"8b1a608d7fce9c091fb9120d4e301565034a75ca4502be800fb3ea13b20d0e06"}
Sep 18 20:05:54.426574 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:54.426554    2578 generic.go:356] "Generic (PLEG): container finished" podID="8c76ea29-391d-4b06-a762-66c2cf0b2101" containerID="f76c82975dc2fdf25d5c22693834479393d4f41d06a28adbc2a8154d563fca35" exitCode=0
Sep 18 20:05:54.426674 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:54.426575    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8" event={"ID":"8c76ea29-391d-4b06-a762-66c2cf0b2101","Type":"ContainerDied","Data":"f76c82975dc2fdf25d5c22693834479393d4f41d06a28adbc2a8154d563fca35"}
Sep 18 20:05:55.433414 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:55.433378    2578 generic.go:356] "Generic (PLEG): container finished" podID="e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1" containerID="8ece1a67620fb93a2bb61d3cba8337f412bca1f09923236cdcb650c9cff5bda8" exitCode=0
Sep 18 20:05:55.433798 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:55.433475    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47" event={"ID":"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1","Type":"ContainerDied","Data":"8ece1a67620fb93a2bb61d3cba8337f412bca1f09923236cdcb650c9cff5bda8"}
Sep 18 20:05:55.435505 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:55.435482    2578 generic.go:356] "Generic (PLEG): container finished" podID="8c76ea29-391d-4b06-a762-66c2cf0b2101" containerID="1c48eb9100340a8489d93f882b8338d20daf76aba541e3dd2e0949e544831354" exitCode=0
Sep 18 20:05:55.435637 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:55.435563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8" event={"ID":"8c76ea29-391d-4b06-a762-66c2cf0b2101","Type":"ContainerDied","Data":"1c48eb9100340a8489d93f882b8338d20daf76aba541e3dd2e0949e544831354"}
Sep 18 20:05:55.437599 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:55.437578    2578 generic.go:356] "Generic (PLEG): container finished" podID="cc7c0dcb-8057-46f2-9768-2ef68036cbf4" containerID="b13711dc29551bb9ad450f37a6f8997417215548144d6ba73bcb66724b162367" exitCode=0
Sep 18 20:05:55.437700 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:55.437600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd" event={"ID":"cc7c0dcb-8057-46f2-9768-2ef68036cbf4","Type":"ContainerDied","Data":"b13711dc29551bb9ad450f37a6f8997417215548144d6ba73bcb66724b162367"}
Sep 18 20:05:56.591972 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.591952    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"
Sep 18 20:05:56.594719 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.594699    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"
Sep 18 20:05:56.625017 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.624992    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"
Sep 18 20:05:56.633127 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.633107    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pqvf\" (UniqueName: \"kubernetes.io/projected/8c76ea29-391d-4b06-a762-66c2cf0b2101-kube-api-access-8pqvf\") pod \"8c76ea29-391d-4b06-a762-66c2cf0b2101\" (UID: \"8c76ea29-391d-4b06-a762-66c2cf0b2101\") "
Sep 18 20:05:56.633226 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.633172    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c76ea29-391d-4b06-a762-66c2cf0b2101-util\") pod \"8c76ea29-391d-4b06-a762-66c2cf0b2101\" (UID: \"8c76ea29-391d-4b06-a762-66c2cf0b2101\") "
Sep 18 20:05:56.633226 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.633202    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-bundle\") pod \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\" (UID: \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\") "
Sep 18 20:05:56.633321 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.633223    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-util\") pod \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\" (UID: \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\") "
Sep 18 20:05:56.633321 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.633254    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hvxg\" (UniqueName: \"kubernetes.io/projected/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-kube-api-access-9hvxg\") pod \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\" (UID: \"cc7c0dcb-8057-46f2-9768-2ef68036cbf4\") "
Sep 18 20:05:56.633321 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.633305    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c76ea29-391d-4b06-a762-66c2cf0b2101-bundle\") pod \"8c76ea29-391d-4b06-a762-66c2cf0b2101\" (UID: \"8c76ea29-391d-4b06-a762-66c2cf0b2101\") "
Sep 18 20:05:56.633984 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.633944    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76ea29-391d-4b06-a762-66c2cf0b2101-bundle" (OuterVolumeSpecName: "bundle") pod "8c76ea29-391d-4b06-a762-66c2cf0b2101" (UID: "8c76ea29-391d-4b06-a762-66c2cf0b2101"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:56.634076 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.634055    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-bundle" (OuterVolumeSpecName: "bundle") pod "cc7c0dcb-8057-46f2-9768-2ef68036cbf4" (UID: "cc7c0dcb-8057-46f2-9768-2ef68036cbf4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:56.635364 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.635339    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c76ea29-391d-4b06-a762-66c2cf0b2101-kube-api-access-8pqvf" (OuterVolumeSpecName: "kube-api-access-8pqvf") pod "8c76ea29-391d-4b06-a762-66c2cf0b2101" (UID: "8c76ea29-391d-4b06-a762-66c2cf0b2101"). InnerVolumeSpecName "kube-api-access-8pqvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:05:56.635436 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.635359    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-kube-api-access-9hvxg" (OuterVolumeSpecName: "kube-api-access-9hvxg") pod "cc7c0dcb-8057-46f2-9768-2ef68036cbf4" (UID: "cc7c0dcb-8057-46f2-9768-2ef68036cbf4"). InnerVolumeSpecName "kube-api-access-9hvxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:05:56.639379 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.639358    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-util" (OuterVolumeSpecName: "util") pod "cc7c0dcb-8057-46f2-9768-2ef68036cbf4" (UID: "cc7c0dcb-8057-46f2-9768-2ef68036cbf4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:56.642544 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.642520    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c76ea29-391d-4b06-a762-66c2cf0b2101-util" (OuterVolumeSpecName: "util") pod "8c76ea29-391d-4b06-a762-66c2cf0b2101" (UID: "8c76ea29-391d-4b06-a762-66c2cf0b2101"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:56.734487 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.734460    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-util\") pod \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\" (UID: \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\") "
Sep 18 20:05:56.734639 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.734530    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-bundle\") pod \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\" (UID: \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\") "
Sep 18 20:05:56.734639 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.734558    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pdzx\" (UniqueName: \"kubernetes.io/projected/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-kube-api-access-9pdzx\") pod \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\" (UID: \"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1\") "
Sep 18 20:05:56.734764 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.734709    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c76ea29-391d-4b06-a762-66c2cf0b2101-util\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:05:56.734764 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.734726    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-bundle\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:05:56.734764 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.734737    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-util\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:05:56.734764 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.734749    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hvxg\" (UniqueName: \"kubernetes.io/projected/cc7c0dcb-8057-46f2-9768-2ef68036cbf4-kube-api-access-9hvxg\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:05:56.734764 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.734763    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c76ea29-391d-4b06-a762-66c2cf0b2101-bundle\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:05:56.735028 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.734776    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pqvf\" (UniqueName: \"kubernetes.io/projected/8c76ea29-391d-4b06-a762-66c2cf0b2101-kube-api-access-8pqvf\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:05:56.735082 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.735050    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-bundle" (OuterVolumeSpecName: "bundle") pod "e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1" (UID: "e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:56.736579 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.736555    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-kube-api-access-9pdzx" (OuterVolumeSpecName: "kube-api-access-9pdzx") pod "e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1" (UID: "e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1"). InnerVolumeSpecName "kube-api-access-9pdzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:05:56.740082 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.740051    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-util" (OuterVolumeSpecName: "util") pod "e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1" (UID: "e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:56.835232 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.835190    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-bundle\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:05:56.835232 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.835228    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pdzx\" (UniqueName: \"kubernetes.io/projected/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-kube-api-access-9pdzx\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:05:56.835232 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:56.835238    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1-util\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:05:57.447766 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:57.447739    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd"
Sep 18 20:05:57.447766 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:57.447750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847allrwd" event={"ID":"cc7c0dcb-8057-46f2-9768-2ef68036cbf4","Type":"ContainerDied","Data":"9cc0d57f5a8735deb1069452de67440b57ca83120147a03026b439bdc2c05e55"}
Sep 18 20:05:57.448030 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:57.447793    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cc0d57f5a8735deb1069452de67440b57ca83120147a03026b439bdc2c05e55"
Sep 18 20:05:57.449709 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:57.449682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47" event={"ID":"e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1","Type":"ContainerDied","Data":"5f762744e62a393ebf53da27da15d81e8b859c7dd6622a21a8f958f3cbc63309"}
Sep 18 20:05:57.449709 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:57.449704    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lwr47"
Sep 18 20:05:57.449709 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:57.449721    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f762744e62a393ebf53da27da15d81e8b859c7dd6622a21a8f958f3cbc63309"
Sep 18 20:05:57.451688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:57.451668    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8" event={"ID":"8c76ea29-391d-4b06-a762-66c2cf0b2101","Type":"ContainerDied","Data":"92fd0b8a8c96eb50acca9a76fb7009605c7aabfae6d011a11f60304e258f9d58"}
Sep 18 20:05:57.451796 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:57.451703    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92fd0b8a8c96eb50acca9a76fb7009605c7aabfae6d011a11f60304e258f9d58"
Sep 18 20:05:57.451796 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:05:57.451712    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6khlg8"
Sep 18 20:06:03.603608 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:03.603573    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:05.935696 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.935664    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"]
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.935904    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c76ea29-391d-4b06-a762-66c2cf0b2101" containerName="extract"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.935914    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c76ea29-391d-4b06-a762-66c2cf0b2101" containerName="extract"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.935922    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1" containerName="util"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.935927    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1" containerName="util"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.935949    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1" containerName="extract"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.935955    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1" containerName="extract"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.935967    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1" containerName="pull"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.935972    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1" containerName="pull"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.935983    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7c0dcb-8057-46f2-9768-2ef68036cbf4" containerName="util"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.935988    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7c0dcb-8057-46f2-9768-2ef68036cbf4" containerName="util"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.935995    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7c0dcb-8057-46f2-9768-2ef68036cbf4" containerName="extract"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.936001    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7c0dcb-8057-46f2-9768-2ef68036cbf4" containerName="extract"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.936006    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c76ea29-391d-4b06-a762-66c2cf0b2101" containerName="util"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.936011    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c76ea29-391d-4b06-a762-66c2cf0b2101" containerName="util"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.936017    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c76ea29-391d-4b06-a762-66c2cf0b2101" containerName="pull"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.936021    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c76ea29-391d-4b06-a762-66c2cf0b2101" containerName="pull"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.936028    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7c0dcb-8057-46f2-9768-2ef68036cbf4" containerName="pull"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.936034    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7c0dcb-8057-46f2-9768-2ef68036cbf4" containerName="pull"
Sep 18 20:06:05.936075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.936080    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c76ea29-391d-4b06-a762-66c2cf0b2101" containerName="extract"
Sep 18 20:06:05.936599 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.936087    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7c0dcb-8057-46f2-9768-2ef68036cbf4" containerName="extract"
Sep 18 20:06:05.936599 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.936095    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e96ae9aa-3fab-4ec8-a0be-d501e1fc3bf1" containerName="extract"
Sep 18 20:06:05.940114 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.940097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"
Sep 18 20:06:05.943177 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.943156    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 20:06:05.943985 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.943968    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-p6x8d\""
Sep 18 20:06:05.944043 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.943972    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 20:06:05.959917 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:05.959894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"]
Sep 18 20:06:06.006174 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:06.006145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95mq2\" (UniqueName: \"kubernetes.io/projected/ac8005a2-21bf-426c-a732-81a866708958-kube-api-access-95mq2\") pod \"limitador-operator-controller-manager-74495c69f8-7n7c7\" (UID: \"ac8005a2-21bf-426c-a732-81a866708958\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"
Sep 18 20:06:06.107140 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:06.107109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mq2\" (UniqueName: \"kubernetes.io/projected/ac8005a2-21bf-426c-a732-81a866708958-kube-api-access-95mq2\") pod \"limitador-operator-controller-manager-74495c69f8-7n7c7\" (UID: \"ac8005a2-21bf-426c-a732-81a866708958\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"
Sep 18 20:06:06.130162 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:06.130132    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mq2\" (UniqueName: \"kubernetes.io/projected/ac8005a2-21bf-426c-a732-81a866708958-kube-api-access-95mq2\") pod \"limitador-operator-controller-manager-74495c69f8-7n7c7\" (UID: \"ac8005a2-21bf-426c-a732-81a866708958\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"
Sep 18 20:06:06.249983 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:06.249957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"
Sep 18 20:06:06.546522 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:06.546500    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"]
Sep 18 20:06:06.548910 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:06:06.548884    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8005a2_21bf_426c_a732_81a866708958.slice/crio-f50131d34430337a41a0df2984638e67009b6af2d6deb15275787d942e2b123d WatchSource:0}: Error finding container f50131d34430337a41a0df2984638e67009b6af2d6deb15275787d942e2b123d: Status 404 returned error can't find the container with id f50131d34430337a41a0df2984638e67009b6af2d6deb15275787d942e2b123d
Sep 18 20:06:06.659742 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:06.659714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:07.496344 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:07.496304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7" event={"ID":"ac8005a2-21bf-426c-a732-81a866708958","Type":"ContainerStarted","Data":"f50131d34430337a41a0df2984638e67009b6af2d6deb15275787d942e2b123d"}
Sep 18 20:06:09.131232 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:09.131194    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:09.488647 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:09.488612    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:09.506546 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:09.506514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7" event={"ID":"ac8005a2-21bf-426c-a732-81a866708958","Type":"ContainerStarted","Data":"ff6df60c23941fb3f53b87bd98e52e38823133fdc63e4c524aec46ca338bc371"}
Sep 18 20:06:09.506674 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:09.506642    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"
Sep 18 20:06:09.564121 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:09.564045    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7" podStartSLOduration=1.943567336 podStartE2EDuration="4.564019019s" totalImagesPullingTime="2.620451683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:09.561671648 +0000 UTC m=+571.400135604" watchObservedRunningTime="2026-09-18 20:06:09.564019019 +0000 UTC m=+571.402482971"
Sep 18 20:06:09.666341 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:09.666297    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:09.973565 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:09.973536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:10.241064 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:10.240977    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:10.750538 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:10.750501    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:10.938071 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:10.938040    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:13.399546 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:13.399512    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:13.804036 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:13.804005    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:14.509246 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:14.509214    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:14.703079 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:14.703037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:16.073120 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:16.073086    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-65cnn"]
Sep 18 20:06:16.076251 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:16.076235    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-65cnn"
Sep 18 20:06:16.079369 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:16.079349    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jkd26\""
Sep 18 20:06:16.089529 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:16.089500    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-65cnn"]
Sep 18 20:06:16.182745 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:16.182714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nxlk\" (UniqueName: \"kubernetes.io/projected/328ce793-8384-410d-9569-7adb784fbf13-kube-api-access-9nxlk\") pod \"authorino-operator-76b478bd78-65cnn\" (UID: \"328ce793-8384-410d-9569-7adb784fbf13\") " pod="kuadrant-system/authorino-operator-76b478bd78-65cnn"
Sep 18 20:06:16.283793 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:16.283757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nxlk\" (UniqueName: \"kubernetes.io/projected/328ce793-8384-410d-9569-7adb784fbf13-kube-api-access-9nxlk\") pod \"authorino-operator-76b478bd78-65cnn\" (UID: \"328ce793-8384-410d-9569-7adb784fbf13\") " pod="kuadrant-system/authorino-operator-76b478bd78-65cnn"
Sep 18 20:06:16.293348 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:16.293323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nxlk\" (UniqueName: \"kubernetes.io/projected/328ce793-8384-410d-9569-7adb784fbf13-kube-api-access-9nxlk\") pod \"authorino-operator-76b478bd78-65cnn\" (UID: \"328ce793-8384-410d-9569-7adb784fbf13\") " pod="kuadrant-system/authorino-operator-76b478bd78-65cnn"
Sep 18 20:06:16.385281 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:16.385197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-65cnn"
Sep 18 20:06:16.661284 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:16.661206    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-65cnn"]
Sep 18 20:06:16.664515 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:06:16.664486    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328ce793_8384_410d_9569_7adb784fbf13.slice/crio-86fdc9117fe34243289c10dc81998c2c74cb032aa41c83cc0ea2b08f11090a8d WatchSource:0}: Error finding container 86fdc9117fe34243289c10dc81998c2c74cb032aa41c83cc0ea2b08f11090a8d: Status 404 returned error can't find the container with id 86fdc9117fe34243289c10dc81998c2c74cb032aa41c83cc0ea2b08f11090a8d
Sep 18 20:06:17.414177 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:17.414127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:17.545794 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:17.545754    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-65cnn" event={"ID":"328ce793-8384-410d-9569-7adb784fbf13","Type":"ContainerStarted","Data":"86fdc9117fe34243289c10dc81998c2c74cb032aa41c83cc0ea2b08f11090a8d"}
Sep 18 20:06:17.553451 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:17.553421    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:18.551845 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:18.551813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-65cnn" event={"ID":"328ce793-8384-410d-9569-7adb784fbf13","Type":"ContainerStarted","Data":"e79af8e1e8f107a198e7d03b3a334d3cd7dfc165f64e0fc46683be981fc0571d"}
Sep 18 20:06:18.552259 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:18.551884    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-65cnn"
Sep 18 20:06:18.588757 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:18.588707    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-65cnn" podStartSLOduration=1.126803549 podStartE2EDuration="2.588690822s" totalImagesPullingTime="1.461887273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:18.588487754 +0000 UTC m=+580.426951707" watchObservedRunningTime="2026-09-18 20:06:18.588690822 +0000 UTC m=+580.427154774"
Sep 18 20:06:18.886507 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:18.886412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:18.997060 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:18.997016    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:20.473541 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:20.473509    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:20.514394 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:20.514360    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"
Sep 18 20:06:20.700638 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:20.700595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:25.799618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:25.799553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:25.882328 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:25.882283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:26.413362 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:26.413326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:26.527631 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:26.527593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:27.023717 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:27.023676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:27.133331 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:27.133295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:29.559026 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:29.558998    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-65cnn"
Sep 18 20:06:31.428228 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.428195    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"]
Sep 18 20:06:31.428675 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.428454    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7" podUID="ac8005a2-21bf-426c-a732-81a866708958" containerName="manager" containerID="cri-o://ff6df60c23941fb3f53b87bd98e52e38823133fdc63e4c524aec46ca338bc371" gracePeriod=2
Sep 18 20:06:31.436159 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.436134    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"]
Sep 18 20:06:31.478203 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.478174    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lcvw"]
Sep 18 20:06:31.478450 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.478439    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac8005a2-21bf-426c-a732-81a866708958" containerName="manager"
Sep 18 20:06:31.478498 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.478453    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac8005a2-21bf-426c-a732-81a866708958" containerName="manager"
Sep 18 20:06:31.478532 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.478499    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac8005a2-21bf-426c-a732-81a866708958" containerName="manager"
Sep 18 20:06:31.481354 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.481339    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lcvw"
Sep 18 20:06:31.491216 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:06:31.491177    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7n7c7\" is forbidden: User \"system:node:ip-10-0-132-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-172.ec2.internal' and this object" podUID="ac8005a2-21bf-426c-a732-81a866708958" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"
Sep 18 20:06:31.497062 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.497040    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lcvw"]
Sep 18 20:06:31.607194 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.607159    2578 generic.go:356] "Generic (PLEG): container finished" podID="ac8005a2-21bf-426c-a732-81a866708958" containerID="ff6df60c23941fb3f53b87bd98e52e38823133fdc63e4c524aec46ca338bc371" exitCode=0
Sep 18 20:06:31.612487 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.612463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h9mz\" (UniqueName: \"kubernetes.io/projected/076f3e26-3618-4db3-84ff-6c07b5d336b1-kube-api-access-9h9mz\") pod \"limitador-operator-controller-manager-74495c69f8-9lcvw\" (UID: \"076f3e26-3618-4db3-84ff-6c07b5d336b1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lcvw"
Sep 18 20:06:31.674530 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.674507    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"
Sep 18 20:06:31.677367 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:06:31.677344    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7n7c7\" is forbidden: User \"system:node:ip-10-0-132-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-172.ec2.internal' and this object" podUID="ac8005a2-21bf-426c-a732-81a866708958" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"
Sep 18 20:06:31.713709 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.713684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h9mz\" (UniqueName: \"kubernetes.io/projected/076f3e26-3618-4db3-84ff-6c07b5d336b1-kube-api-access-9h9mz\") pod \"limitador-operator-controller-manager-74495c69f8-9lcvw\" (UID: \"076f3e26-3618-4db3-84ff-6c07b5d336b1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lcvw"
Sep 18 20:06:31.753278 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.753249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h9mz\" (UniqueName: \"kubernetes.io/projected/076f3e26-3618-4db3-84ff-6c07b5d336b1-kube-api-access-9h9mz\") pod \"limitador-operator-controller-manager-74495c69f8-9lcvw\" (UID: \"076f3e26-3618-4db3-84ff-6c07b5d336b1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lcvw"
Sep 18 20:06:31.814975 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.814928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95mq2\" (UniqueName: \"kubernetes.io/projected/ac8005a2-21bf-426c-a732-81a866708958-kube-api-access-95mq2\") pod \"ac8005a2-21bf-426c-a732-81a866708958\" (UID: \"ac8005a2-21bf-426c-a732-81a866708958\") "
Sep 18 20:06:31.816999 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.816969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8005a2-21bf-426c-a732-81a866708958-kube-api-access-95mq2" (OuterVolumeSpecName: "kube-api-access-95mq2") pod "ac8005a2-21bf-426c-a732-81a866708958" (UID: "ac8005a2-21bf-426c-a732-81a866708958"). InnerVolumeSpecName "kube-api-access-95mq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:31.817998 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.817980    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lcvw"
Sep 18 20:06:31.916451 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:31.916416    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95mq2\" (UniqueName: \"kubernetes.io/projected/ac8005a2-21bf-426c-a732-81a866708958-kube-api-access-95mq2\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:06:32.133603 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:32.133518    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lcvw"]
Sep 18 20:06:32.137225 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:06:32.137200    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076f3e26_3618_4db3_84ff_6c07b5d336b1.slice/crio-d4390d97b1f31d34771c6d3b2e56d6c6ee5b59a6fcc62026f3f4488f79125957 WatchSource:0}: Error finding container d4390d97b1f31d34771c6d3b2e56d6c6ee5b59a6fcc62026f3f4488f79125957: Status 404 returned error can't find the container with id d4390d97b1f31d34771c6d3b2e56d6c6ee5b59a6fcc62026f3f4488f79125957
Sep 18 20:06:32.317481 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:32.317448    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac8005a2-21bf-426c-a732-81a866708958" path="/var/lib/kubelet/pods/ac8005a2-21bf-426c-a732-81a866708958/volumes"
Sep 18 20:06:32.613260 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:32.613227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lcvw" event={"ID":"076f3e26-3618-4db3-84ff-6c07b5d336b1","Type":"ContainerStarted","Data":"f1b64c7078a48bdc7b95cce1677e4aefbcb1d289d0ea7daadd11fa73fbbae4bd"}
Sep 18 20:06:32.613670 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:32.613279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lcvw" event={"ID":"076f3e26-3618-4db3-84ff-6c07b5d336b1","Type":"ContainerStarted","Data":"d4390d97b1f31d34771c6d3b2e56d6c6ee5b59a6fcc62026f3f4488f79125957"}
Sep 18 20:06:32.613670 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:32.613331    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lcvw"
Sep 18 20:06:32.614699 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:32.614680    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7n7c7"
Sep 18 20:06:32.614818 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:32.614694    2578 scope.go:117] "RemoveContainer" containerID="ff6df60c23941fb3f53b87bd98e52e38823133fdc63e4c524aec46ca338bc371"
Sep 18 20:06:32.669860 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:32.669813    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lcvw" podStartSLOduration=1.6697982470000001 podStartE2EDuration="1.669798247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:32.668891509 +0000 UTC m=+594.507355462" watchObservedRunningTime="2026-09-18 20:06:32.669798247 +0000 UTC m=+594.508262196"
Sep 18 20:06:35.810816 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:35.810779    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:35.926230 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:35.926188    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:36.404451 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:36.404418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:36.509412 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:36.509375    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:41.588847 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:41.588809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:41.690577 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:41.690549    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:43.623337 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:43.623303    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9lcvw"
Sep 18 20:06:44.503983 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:44.503948    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:44.638183 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:44.638148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:46.542668 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:46.542635    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:46.631853 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:46.631823    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:49.995104 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:49.995061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:50.116731 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:50.116694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:50.216637 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:50.216601    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:58.834172 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:58.834100    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:58.931888 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:06:58.931848    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:01.700787 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:01.700753    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:01.904428 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:01.904384    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:04.306813 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:04.306775    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:04.464109 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:04.464080    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:04.563108 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:04.563036    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:05.303706 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:05.303667    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hwvnq"]
Sep 18 20:07:05.306961 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:05.306923    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hwvnq"
Sep 18 20:07:05.309642 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:05.309624    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nhv9b\""
Sep 18 20:07:05.333874 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:05.333850    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hwvnq"]
Sep 18 20:07:05.366385 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:05.366356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:05.373356 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:05.373332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55zcz\" (UniqueName: \"kubernetes.io/projected/5ace6f14-7ed5-4c97-b11a-99ba62e213e5-kube-api-access-55zcz\") pod \"authorino-7bcfd8f9cd-hwvnq\" (UID: \"5ace6f14-7ed5-4c97-b11a-99ba62e213e5\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hwvnq"
Sep 18 20:07:05.468639 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:05.468603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:05.474552 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:05.474521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55zcz\" (UniqueName: \"kubernetes.io/projected/5ace6f14-7ed5-4c97-b11a-99ba62e213e5-kube-api-access-55zcz\") pod \"authorino-7bcfd8f9cd-hwvnq\" (UID: \"5ace6f14-7ed5-4c97-b11a-99ba62e213e5\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hwvnq"
Sep 18 20:07:05.484160 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:05.484126    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55zcz\" (UniqueName: \"kubernetes.io/projected/5ace6f14-7ed5-4c97-b11a-99ba62e213e5-kube-api-access-55zcz\") pod \"authorino-7bcfd8f9cd-hwvnq\" (UID: \"5ace6f14-7ed5-4c97-b11a-99ba62e213e5\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hwvnq"
Sep 18 20:07:05.615719 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:05.615649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hwvnq"
Sep 18 20:07:05.885245 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:05.885131    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hwvnq"]
Sep 18 20:07:05.887616 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:07:05.887589    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ace6f14_7ed5_4c97_b11a_99ba62e213e5.slice/crio-3f2fd5460dcee1a0509554944980c379c37e62fce279c620aa578903e2781001 WatchSource:0}: Error finding container 3f2fd5460dcee1a0509554944980c379c37e62fce279c620aa578903e2781001: Status 404 returned error can't find the container with id 3f2fd5460dcee1a0509554944980c379c37e62fce279c620aa578903e2781001
Sep 18 20:07:06.760629 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:06.760601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hwvnq" event={"ID":"5ace6f14-7ed5-4c97-b11a-99ba62e213e5","Type":"ContainerStarted","Data":"3f2fd5460dcee1a0509554944980c379c37e62fce279c620aa578903e2781001"}
Sep 18 20:07:08.573302 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:08.573268    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hwvnq"]
Sep 18 20:07:08.771664 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:08.771621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hwvnq" event={"ID":"5ace6f14-7ed5-4c97-b11a-99ba62e213e5","Type":"ContainerStarted","Data":"2561abd08e14fe1dbd018849e71fafdf513b301beb568b6c4a8eb94bfd964ebd"}
Sep 18 20:07:08.793242 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:08.793189    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-hwvnq" podStartSLOduration=1.7632416260000001 podStartE2EDuration="3.793174603s" totalImagesPullingTime="2.029932977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:08.791163841 +0000 UTC m=+630.629627793" watchObservedRunningTime="2026-09-18 20:07:08.793174603 +0000 UTC m=+630.631638554"
Sep 18 20:07:09.776533 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:09.776489    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-hwvnq" podUID="5ace6f14-7ed5-4c97-b11a-99ba62e213e5" containerName="authorino" containerID="cri-o://2561abd08e14fe1dbd018849e71fafdf513b301beb568b6c4a8eb94bfd964ebd" gracePeriod=30
Sep 18 20:07:10.022273 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:10.022245    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hwvnq"
Sep 18 20:07:10.111578 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:10.111495    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55zcz\" (UniqueName: \"kubernetes.io/projected/5ace6f14-7ed5-4c97-b11a-99ba62e213e5-kube-api-access-55zcz\") pod \"5ace6f14-7ed5-4c97-b11a-99ba62e213e5\" (UID: \"5ace6f14-7ed5-4c97-b11a-99ba62e213e5\") "
Sep 18 20:07:10.113565 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:10.113528    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ace6f14-7ed5-4c97-b11a-99ba62e213e5-kube-api-access-55zcz" (OuterVolumeSpecName: "kube-api-access-55zcz") pod "5ace6f14-7ed5-4c97-b11a-99ba62e213e5" (UID: "5ace6f14-7ed5-4c97-b11a-99ba62e213e5"). InnerVolumeSpecName "kube-api-access-55zcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:10.212899 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:10.212869    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55zcz\" (UniqueName: \"kubernetes.io/projected/5ace6f14-7ed5-4c97-b11a-99ba62e213e5-kube-api-access-55zcz\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:07:10.781498 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:10.781465    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ace6f14-7ed5-4c97-b11a-99ba62e213e5" containerID="2561abd08e14fe1dbd018849e71fafdf513b301beb568b6c4a8eb94bfd964ebd" exitCode=0
Sep 18 20:07:10.781902 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:10.781510    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hwvnq"
Sep 18 20:07:10.781902 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:10.781552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hwvnq" event={"ID":"5ace6f14-7ed5-4c97-b11a-99ba62e213e5","Type":"ContainerDied","Data":"2561abd08e14fe1dbd018849e71fafdf513b301beb568b6c4a8eb94bfd964ebd"}
Sep 18 20:07:10.781902 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:10.781597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hwvnq" event={"ID":"5ace6f14-7ed5-4c97-b11a-99ba62e213e5","Type":"ContainerDied","Data":"3f2fd5460dcee1a0509554944980c379c37e62fce279c620aa578903e2781001"}
Sep 18 20:07:10.781902 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:10.781617    2578 scope.go:117] "RemoveContainer" containerID="2561abd08e14fe1dbd018849e71fafdf513b301beb568b6c4a8eb94bfd964ebd"
Sep 18 20:07:10.789622 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:10.789606    2578 scope.go:117] "RemoveContainer" containerID="2561abd08e14fe1dbd018849e71fafdf513b301beb568b6c4a8eb94bfd964ebd"
Sep 18 20:07:10.789852 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:07:10.789833    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2561abd08e14fe1dbd018849e71fafdf513b301beb568b6c4a8eb94bfd964ebd\": container with ID starting with 2561abd08e14fe1dbd018849e71fafdf513b301beb568b6c4a8eb94bfd964ebd not found: ID does not exist" containerID="2561abd08e14fe1dbd018849e71fafdf513b301beb568b6c4a8eb94bfd964ebd"
Sep 18 20:07:10.789917 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:10.789862    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2561abd08e14fe1dbd018849e71fafdf513b301beb568b6c4a8eb94bfd964ebd"} err="failed to get container status \"2561abd08e14fe1dbd018849e71fafdf513b301beb568b6c4a8eb94bfd964ebd\": rpc error: code = NotFound desc = could not find container \"2561abd08e14fe1dbd018849e71fafdf513b301beb568b6c4a8eb94bfd964ebd\": container with ID starting with 2561abd08e14fe1dbd018849e71fafdf513b301beb568b6c4a8eb94bfd964ebd not found: ID does not exist"
Sep 18 20:07:10.797221 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:10.797200    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hwvnq"]
Sep 18 20:07:10.801218 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:10.801200    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hwvnq"]
Sep 18 20:07:12.317145 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:12.317113    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ace6f14-7ed5-4c97-b11a-99ba62e213e5" path="/var/lib/kubelet/pods/5ace6f14-7ed5-4c97-b11a-99ba62e213e5/volumes"
Sep 18 20:07:20.451653 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:20.451614    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:20.541586 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:20.541552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:20.844555 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:20.844524    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph"]
Sep 18 20:07:20.844888 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:20.844872    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ace6f14-7ed5-4c97-b11a-99ba62e213e5" containerName="authorino"
Sep 18 20:07:20.845015 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:20.844891    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ace6f14-7ed5-4c97-b11a-99ba62e213e5" containerName="authorino"
Sep 18 20:07:20.845015 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:20.844985    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ace6f14-7ed5-4c97-b11a-99ba62e213e5" containerName="authorino"
Sep 18 20:07:20.849191 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:20.849169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph"
Sep 18 20:07:20.851356 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:20.851335    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 20:07:20.851466 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:20.851399    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vwz2g\""
Sep 18 20:07:20.851989 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:20.851972    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 18 20:07:20.852070 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:20.851997    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 20:07:20.856438 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:20.856416    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph"]
Sep 18 20:07:20.885741 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:20.885714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:21.004014 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:21.003980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v2tl\" (UniqueName: \"kubernetes.io/projected/f376959c-a49d-405b-afcb-c8deec84a921-kube-api-access-4v2tl\") pod \"postgres-69fb4594cf-gf5ph\" (UID: \"f376959c-a49d-405b-afcb-c8deec84a921\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph"
Sep 18 20:07:21.004169 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:21.004031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f376959c-a49d-405b-afcb-c8deec84a921-data\") pod \"postgres-69fb4594cf-gf5ph\" (UID: \"f376959c-a49d-405b-afcb-c8deec84a921\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph"
Sep 18 20:07:21.104635 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:21.104558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v2tl\" (UniqueName: \"kubernetes.io/projected/f376959c-a49d-405b-afcb-c8deec84a921-kube-api-access-4v2tl\") pod \"postgres-69fb4594cf-gf5ph\" (UID: \"f376959c-a49d-405b-afcb-c8deec84a921\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph"
Sep 18 20:07:21.104635 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:21.104599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f376959c-a49d-405b-afcb-c8deec84a921-data\") pod \"postgres-69fb4594cf-gf5ph\" (UID: \"f376959c-a49d-405b-afcb-c8deec84a921\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph"
Sep 18 20:07:21.104996 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:21.104980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f376959c-a49d-405b-afcb-c8deec84a921-data\") pod \"postgres-69fb4594cf-gf5ph\" (UID: \"f376959c-a49d-405b-afcb-c8deec84a921\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph"
Sep 18 20:07:21.113286 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:21.113266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v2tl\" (UniqueName: \"kubernetes.io/projected/f376959c-a49d-405b-afcb-c8deec84a921-kube-api-access-4v2tl\") pod \"postgres-69fb4594cf-gf5ph\" (UID: \"f376959c-a49d-405b-afcb-c8deec84a921\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph"
Sep 18 20:07:21.159468 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:21.159436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph"
Sep 18 20:07:21.431822 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:21.431661    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph"]
Sep 18 20:07:21.434478 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:07:21.434446    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf376959c_a49d_405b_afcb_c8deec84a921.slice/crio-63f73cb360e05ccc7e9d7a4c03fde5384de2d7c17d07724522cbe98755bdda25 WatchSource:0}: Error finding container 63f73cb360e05ccc7e9d7a4c03fde5384de2d7c17d07724522cbe98755bdda25: Status 404 returned error can't find the container with id 63f73cb360e05ccc7e9d7a4c03fde5384de2d7c17d07724522cbe98755bdda25
Sep 18 20:07:21.829869 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:21.829833    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph" event={"ID":"f376959c-a49d-405b-afcb-c8deec84a921","Type":"ContainerStarted","Data":"63f73cb360e05ccc7e9d7a4c03fde5384de2d7c17d07724522cbe98755bdda25"}
Sep 18 20:07:26.856790 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:26.856753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph" event={"ID":"f376959c-a49d-405b-afcb-c8deec84a921","Type":"ContainerStarted","Data":"6cc74529702b5556a5659e98336cad0c18380fba80e7f7b22014ff0c3c5d25e6"}
Sep 18 20:07:26.857265 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:26.856871    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph"
Sep 18 20:07:26.874780 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:26.874731    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph" podStartSLOduration=2.1273184880000002 podStartE2EDuration="6.874716364s" totalImagesPullingTime="4.747397876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:26.872978395 +0000 UTC m=+648.711442348" watchObservedRunningTime="2026-09-18 20:07:26.874716364 +0000 UTC m=+648.713180316"
Sep 18 20:07:32.889370 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:32.889342    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gf5ph"
Sep 18 20:07:33.212110 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.212017    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:33.428404 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.428354    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-zpr76"]
Sep 18 20:07:33.434944 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.434918    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zpr76"
Sep 18 20:07:33.437622 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.437594    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nhv9b\""
Sep 18 20:07:33.442035 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.442015    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-zpr76"]
Sep 18 20:07:33.547141 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.547108    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:33.603188 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.603155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4cz2\" (UniqueName: \"kubernetes.io/projected/6940968b-80e3-45ce-a318-1af63e256a2a-kube-api-access-s4cz2\") pod \"authorino-557f876f69-zpr76\" (UID: \"6940968b-80e3-45ce-a318-1af63e256a2a\") " pod="kuadrant-system/authorino-557f876f69-zpr76"
Sep 18 20:07:33.641488 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.641458    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zpr76"]
Sep 18 20:07:33.641670 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:07:33.641651    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-s4cz2], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-zpr76" podUID="6940968b-80e3-45ce-a318-1af63e256a2a"
Sep 18 20:07:33.694178 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.694144    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-659975bbf5-vhnrs"]
Sep 18 20:07:33.697836 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.697815    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-659975bbf5-vhnrs"
Sep 18 20:07:33.700103 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.700085    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 20:07:33.703761 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.703735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4cz2\" (UniqueName: \"kubernetes.io/projected/6940968b-80e3-45ce-a318-1af63e256a2a-kube-api-access-s4cz2\") pod \"authorino-557f876f69-zpr76\" (UID: \"6940968b-80e3-45ce-a318-1af63e256a2a\") " pod="kuadrant-system/authorino-557f876f69-zpr76"
Sep 18 20:07:33.708222 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.708202    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-659975bbf5-vhnrs"]
Sep 18 20:07:33.717449 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.717428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4cz2\" (UniqueName: \"kubernetes.io/projected/6940968b-80e3-45ce-a318-1af63e256a2a-kube-api-access-s4cz2\") pod \"authorino-557f876f69-zpr76\" (UID: \"6940968b-80e3-45ce-a318-1af63e256a2a\") " pod="kuadrant-system/authorino-557f876f69-zpr76"
Sep 18 20:07:33.804346 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.804274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a0c8f85-c704-434a-ad7d-2e36790e4fa5-tls-cert\") pod \"authorino-659975bbf5-vhnrs\" (UID: \"7a0c8f85-c704-434a-ad7d-2e36790e4fa5\") " pod="kuadrant-system/authorino-659975bbf5-vhnrs"
Sep 18 20:07:33.804346 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.804337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lv8z\" (UniqueName: \"kubernetes.io/projected/7a0c8f85-c704-434a-ad7d-2e36790e4fa5-kube-api-access-6lv8z\") pod \"authorino-659975bbf5-vhnrs\" (UID: \"7a0c8f85-c704-434a-ad7d-2e36790e4fa5\") " pod="kuadrant-system/authorino-659975bbf5-vhnrs"
Sep 18 20:07:33.885773 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.885744    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zpr76"
Sep 18 20:07:33.893150 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.893125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zpr76"
Sep 18 20:07:33.904663 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.904643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a0c8f85-c704-434a-ad7d-2e36790e4fa5-tls-cert\") pod \"authorino-659975bbf5-vhnrs\" (UID: \"7a0c8f85-c704-434a-ad7d-2e36790e4fa5\") " pod="kuadrant-system/authorino-659975bbf5-vhnrs"
Sep 18 20:07:33.904747 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.904689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lv8z\" (UniqueName: \"kubernetes.io/projected/7a0c8f85-c704-434a-ad7d-2e36790e4fa5-kube-api-access-6lv8z\") pod \"authorino-659975bbf5-vhnrs\" (UID: \"7a0c8f85-c704-434a-ad7d-2e36790e4fa5\") " pod="kuadrant-system/authorino-659975bbf5-vhnrs"
Sep 18 20:07:33.907134 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.907113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a0c8f85-c704-434a-ad7d-2e36790e4fa5-tls-cert\") pod \"authorino-659975bbf5-vhnrs\" (UID: \"7a0c8f85-c704-434a-ad7d-2e36790e4fa5\") " pod="kuadrant-system/authorino-659975bbf5-vhnrs"
Sep 18 20:07:33.913074 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:33.913055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lv8z\" (UniqueName: \"kubernetes.io/projected/7a0c8f85-c704-434a-ad7d-2e36790e4fa5-kube-api-access-6lv8z\") pod \"authorino-659975bbf5-vhnrs\" (UID: \"7a0c8f85-c704-434a-ad7d-2e36790e4fa5\") " pod="kuadrant-system/authorino-659975bbf5-vhnrs"
Sep 18 20:07:34.004976 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:34.004950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4cz2\" (UniqueName: \"kubernetes.io/projected/6940968b-80e3-45ce-a318-1af63e256a2a-kube-api-access-s4cz2\") pod \"6940968b-80e3-45ce-a318-1af63e256a2a\" (UID: \"6940968b-80e3-45ce-a318-1af63e256a2a\") "
Sep 18 20:07:34.006395 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:34.006370    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-659975bbf5-vhnrs"
Sep 18 20:07:34.006946 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:34.006909    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6940968b-80e3-45ce-a318-1af63e256a2a-kube-api-access-s4cz2" (OuterVolumeSpecName: "kube-api-access-s4cz2") pod "6940968b-80e3-45ce-a318-1af63e256a2a" (UID: "6940968b-80e3-45ce-a318-1af63e256a2a"). InnerVolumeSpecName "kube-api-access-s4cz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:34.105794 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:34.105761    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4cz2\" (UniqueName: \"kubernetes.io/projected/6940968b-80e3-45ce-a318-1af63e256a2a-kube-api-access-s4cz2\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:07:34.278321 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:34.278245    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-659975bbf5-vhnrs"]
Sep 18 20:07:34.281102 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:07:34.281075    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a0c8f85_c704_434a_ad7d_2e36790e4fa5.slice/crio-a7fd964aee2dfa3bf4bdae06b2a3cb8dfb2a8a4cf18e102d0742669550ba3cca WatchSource:0}: Error finding container a7fd964aee2dfa3bf4bdae06b2a3cb8dfb2a8a4cf18e102d0742669550ba3cca: Status 404 returned error can't find the container with id a7fd964aee2dfa3bf4bdae06b2a3cb8dfb2a8a4cf18e102d0742669550ba3cca
Sep 18 20:07:34.891070 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:34.891042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zpr76"
Sep 18 20:07:34.891236 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:34.891042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-659975bbf5-vhnrs" event={"ID":"7a0c8f85-c704-434a-ad7d-2e36790e4fa5","Type":"ContainerStarted","Data":"a7fd964aee2dfa3bf4bdae06b2a3cb8dfb2a8a4cf18e102d0742669550ba3cca"}
Sep 18 20:07:34.918491 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:34.918463    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zpr76"]
Sep 18 20:07:34.924186 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:34.924161    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-zpr76"]
Sep 18 20:07:35.896863 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:35.896823    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-659975bbf5-vhnrs" event={"ID":"7a0c8f85-c704-434a-ad7d-2e36790e4fa5","Type":"ContainerStarted","Data":"2be0340f730a046901045d19cbf7e581bc6f69ee539238df42bdc98a0fcf74dc"}
Sep 18 20:07:35.913681 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:35.913638    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-659975bbf5-vhnrs" podStartSLOduration=1.990528882 podStartE2EDuration="2.913622063s" totalImagesPullingTime="923.093181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:35.912966796 +0000 UTC m=+657.751430750" watchObservedRunningTime="2026-09-18 20:07:35.913622063 +0000 UTC m=+657.752086014"
Sep 18 20:07:36.317668 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:36.317634    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6940968b-80e3-45ce-a318-1af63e256a2a" path="/var/lib/kubelet/pods/6940968b-80e3-45ce-a318-1af63e256a2a/volumes"
Sep 18 20:07:40.678103 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:40.678021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:40.777974 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:40.777924    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:40.867395 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:40.867362    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:40.943110 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:40.943024    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:44.497324 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:44.497284    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:44.643951 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:07:44.643911    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:14.973045 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:14.973009    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:16.274057 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:16.274019    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:16.485982 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:16.485942    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:16.581306 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:16.581229    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:16.696239 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:16.696209    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:17.123380 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:17.123347    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:17.303497 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:17.303463    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:17.415035 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:17.415000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:18.290628 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:18.290593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:20.182499 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:20.182462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:28.034517 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:28.034477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:28.127413 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:28.127377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:53.248983 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.248951    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"]
Sep 18 20:08:53.252217 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.252201    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.254715 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.254689    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c5rk2\""
Sep 18 20:08:53.254827 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.254734    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 20:08:53.255364 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.255345    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 20:08:53.255430 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.255405    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 18 20:08:53.272292 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.272270    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"]
Sep 18 20:08:53.289897 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.289865    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:08:53.335710 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.335671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb490-c889-4d39-8adf-85c2bae6c12b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.335870 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.335743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb490-c889-4d39-8adf-85c2bae6c12b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.335870 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.335802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aeeb490-c889-4d39-8adf-85c2bae6c12b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.335984 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.335869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb490-c889-4d39-8adf-85c2bae6c12b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.335984 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.335897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb490-c889-4d39-8adf-85c2bae6c12b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.335984 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.335959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk7kd\" (UniqueName: \"kubernetes.io/projected/3aeeb490-c889-4d39-8adf-85c2bae6c12b-kube-api-access-rk7kd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.437094 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.437053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aeeb490-c889-4d39-8adf-85c2bae6c12b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.437268 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.437107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb490-c889-4d39-8adf-85c2bae6c12b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.437268 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.437135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb490-c889-4d39-8adf-85c2bae6c12b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.437268 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.437169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk7kd\" (UniqueName: \"kubernetes.io/projected/3aeeb490-c889-4d39-8adf-85c2bae6c12b-kube-api-access-rk7kd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.437268 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.437213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb490-c889-4d39-8adf-85c2bae6c12b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.437472 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.437333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb490-c889-4d39-8adf-85c2bae6c12b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.437619 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.437598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb490-c889-4d39-8adf-85c2bae6c12b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.437619 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.437613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb490-c889-4d39-8adf-85c2bae6c12b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.437727 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.437688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb490-c889-4d39-8adf-85c2bae6c12b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.439531 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.439508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb490-c889-4d39-8adf-85c2bae6c12b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.439708 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.439689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aeeb490-c889-4d39-8adf-85c2bae6c12b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.448202 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.448184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk7kd\" (UniqueName: \"kubernetes.io/projected/3aeeb490-c889-4d39-8adf-85c2bae6c12b-kube-api-access-rk7kd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4\" (UID: \"3aeeb490-c889-4d39-8adf-85c2bae6c12b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.562115 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.562035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:08:53.837058 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.837034    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"]
Sep 18 20:08:53.840140 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:08:53.840111    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aeeb490_c889_4d39_8adf_85c2bae6c12b.slice/crio-5f11cb0d6d99347f82b091bf1197d9cb808e7626571239488259b8b70246e753 WatchSource:0}: Error finding container 5f11cb0d6d99347f82b091bf1197d9cb808e7626571239488259b8b70246e753: Status 404 returned error can't find the container with id 5f11cb0d6d99347f82b091bf1197d9cb808e7626571239488259b8b70246e753
Sep 18 20:08:53.841405 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:53.841384    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:08:54.235948 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:54.235904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4" event={"ID":"3aeeb490-c889-4d39-8adf-85c2bae6c12b","Type":"ContainerStarted","Data":"5f11cb0d6d99347f82b091bf1197d9cb808e7626571239488259b8b70246e753"}
Sep 18 20:08:55.242213 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:55.242174    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4" event={"ID":"3aeeb490-c889-4d39-8adf-85c2bae6c12b","Type":"ContainerStarted","Data":"6d31db8ce6db3bc4a68648cc4052b1fe473a676a89362af99ebe883dd18e4be9"}
Sep 18 20:08:55.266103 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:08:55.266053    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4" podStartSLOduration=1.253920131 podStartE2EDuration="2.266036658s" totalImagesPullingTime="1.012116527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:08:55.264779578 +0000 UTC m=+737.103243529" watchObservedRunningTime="2026-09-18 20:08:55.266036658 +0000 UTC m=+737.104500610"
Sep 18 20:09:03.562175 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:03.562134    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:09:03.562602 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:03.562206    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:09:03.574698 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:03.574667    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:09:04.294245 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:04.294216    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4"
Sep 18 20:09:04.986504 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:04.986468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:14.657543 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:14.657506    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:26.832069 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:26.832027    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"]
Sep 18 20:09:26.835439 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:26.835414    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:26.838854 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:26.838828    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 18 20:09:26.853899 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:26.853870    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"]
Sep 18 20:09:26.914479 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:26.914450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b68fca-6f68-4e8d-8e44-72f731e40741-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:26.914630 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:26.914498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcsn2\" (UniqueName: \"kubernetes.io/projected/73b68fca-6f68-4e8d-8e44-72f731e40741-kube-api-access-dcsn2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:26.914630 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:26.914521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b68fca-6f68-4e8d-8e44-72f731e40741-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:26.914630 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:26.914559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b68fca-6f68-4e8d-8e44-72f731e40741-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:26.914630 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:26.914574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b68fca-6f68-4e8d-8e44-72f731e40741-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:26.914630 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:26.914592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b68fca-6f68-4e8d-8e44-72f731e40741-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:26.945705 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:26.945674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:27.015269 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.015234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b68fca-6f68-4e8d-8e44-72f731e40741-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:27.015437 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.015274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b68fca-6f68-4e8d-8e44-72f731e40741-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:27.015437 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.015298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b68fca-6f68-4e8d-8e44-72f731e40741-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:27.015437 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.015323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b68fca-6f68-4e8d-8e44-72f731e40741-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:27.015437 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.015377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b68fca-6f68-4e8d-8e44-72f731e40741-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:27.015437 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.015425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcsn2\" (UniqueName: \"kubernetes.io/projected/73b68fca-6f68-4e8d-8e44-72f731e40741-kube-api-access-dcsn2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:27.015691 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.015669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b68fca-6f68-4e8d-8e44-72f731e40741-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:27.015746 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.015694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b68fca-6f68-4e8d-8e44-72f731e40741-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:27.015797 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.015776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b68fca-6f68-4e8d-8e44-72f731e40741-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:27.017677 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.017648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b68fca-6f68-4e8d-8e44-72f731e40741-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:27.017835 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.017818    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b68fca-6f68-4e8d-8e44-72f731e40741-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:27.028274 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.028253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcsn2\" (UniqueName: \"kubernetes.io/projected/73b68fca-6f68-4e8d-8e44-72f731e40741-kube-api-access-dcsn2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-gdbqn\" (UID: \"73b68fca-6f68-4e8d-8e44-72f731e40741\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:27.145154 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.145069    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:27.422738 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:27.422711    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"]
Sep 18 20:09:27.424795 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:09:27.424772    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b68fca_6f68_4e8d_8e44_72f731e40741.slice/crio-9a2ef79908cc946860652ee4d7ba56b8ebbd0f2d534d6b77a372b60f3a933bcf WatchSource:0}: Error finding container 9a2ef79908cc946860652ee4d7ba56b8ebbd0f2d534d6b77a372b60f3a933bcf: Status 404 returned error can't find the container with id 9a2ef79908cc946860652ee4d7ba56b8ebbd0f2d534d6b77a372b60f3a933bcf
Sep 18 20:09:28.396548 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:28.396508    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn" event={"ID":"73b68fca-6f68-4e8d-8e44-72f731e40741","Type":"ContainerStarted","Data":"bc4e80d009ca2f458d898047d21408f243b7cff94674d88602aed8081f5e11ae"}
Sep 18 20:09:28.396918 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:28.396577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn" event={"ID":"73b68fca-6f68-4e8d-8e44-72f731e40741","Type":"ContainerStarted","Data":"9a2ef79908cc946860652ee4d7ba56b8ebbd0f2d534d6b77a372b60f3a933bcf"}
Sep 18 20:09:28.423065 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:28.423012    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn" podStartSLOduration=2.202867323 podStartE2EDuration="2.422995802s" totalImagesPullingTime="220.128479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:28.42118893 +0000 UTC m=+770.259652884" watchObservedRunningTime="2026-09-18 20:09:28.422995802 +0000 UTC m=+770.261459753"
Sep 18 20:09:37.145423 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:37.145387    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:37.145798 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:37.145454    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:37.158172 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:37.158137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:37.448138 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:37.448053    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-gdbqn"
Sep 18 20:09:38.681268 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.681234    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"]
Sep 18 20:09:38.684659 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.684642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.693444 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.693418    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 18 20:09:38.790251 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.790221    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"]
Sep 18 20:09:38.807700 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.807667    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:38.815444 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.815417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02e801ca-602a-4eff-8119-00f494312bcc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.815444 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.815447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02e801ca-602a-4eff-8119-00f494312bcc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.815628 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.815479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e801ca-602a-4eff-8119-00f494312bcc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.815628 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.815497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02e801ca-602a-4eff-8119-00f494312bcc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.815628 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.815558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02e801ca-602a-4eff-8119-00f494312bcc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.815628 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.815606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x75n\" (UniqueName: \"kubernetes.io/projected/02e801ca-602a-4eff-8119-00f494312bcc-kube-api-access-4x75n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.916491 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.916456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02e801ca-602a-4eff-8119-00f494312bcc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.916667 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.916498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02e801ca-602a-4eff-8119-00f494312bcc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.916667 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.916632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x75n\" (UniqueName: \"kubernetes.io/projected/02e801ca-602a-4eff-8119-00f494312bcc-kube-api-access-4x75n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.916789 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.916728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02e801ca-602a-4eff-8119-00f494312bcc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.916789 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.916761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02e801ca-602a-4eff-8119-00f494312bcc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.916889 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.916825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e801ca-602a-4eff-8119-00f494312bcc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.917009 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.916988    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02e801ca-602a-4eff-8119-00f494312bcc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.917160 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.917136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02e801ca-602a-4eff-8119-00f494312bcc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.917229 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.917167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02e801ca-602a-4eff-8119-00f494312bcc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.918805 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.918782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02e801ca-602a-4eff-8119-00f494312bcc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.919192 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.919176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02e801ca-602a-4eff-8119-00f494312bcc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.929452 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.929428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x75n\" (UniqueName: \"kubernetes.io/projected/02e801ca-602a-4eff-8119-00f494312bcc-kube-api-access-4x75n\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48\" (UID: \"02e801ca-602a-4eff-8119-00f494312bcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:38.994515 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:38.994487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:39.287574 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:39.287543    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"]
Sep 18 20:09:39.288913 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:09:39.288881    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e801ca_602a_4eff_8119_00f494312bcc.slice/crio-73fa80179e1157db82dce7ccf8ca544008fd9bcc665426a9d3893421dcbd3ebc WatchSource:0}: Error finding container 73fa80179e1157db82dce7ccf8ca544008fd9bcc665426a9d3893421dcbd3ebc: Status 404 returned error can't find the container with id 73fa80179e1157db82dce7ccf8ca544008fd9bcc665426a9d3893421dcbd3ebc
Sep 18 20:09:39.449231 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:39.449195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48" event={"ID":"02e801ca-602a-4eff-8119-00f494312bcc","Type":"ContainerStarted","Data":"73fa80179e1157db82dce7ccf8ca544008fd9bcc665426a9d3893421dcbd3ebc"}
Sep 18 20:09:40.456496 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:40.456460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48" event={"ID":"02e801ca-602a-4eff-8119-00f494312bcc","Type":"ContainerStarted","Data":"de2f0f147d6fdcfaa6606bbaf229c7bd786b94e4a6c32f631af1d3afdefea8c6"}
Sep 18 20:09:40.488115 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:40.488056    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48" podStartSLOduration=2.297388188 podStartE2EDuration="2.488039494s" totalImagesPullingTime="190.651306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:40.487407496 +0000 UTC m=+782.325871450" watchObservedRunningTime="2026-09-18 20:09:40.488039494 +0000 UTC m=+782.326503445"
Sep 18 20:09:45.530676 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:45.530630    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:48.995551 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:48.995519    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:48.995992 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:48.995602    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:49.007911 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:49.007888    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:49.507834 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:49.507799    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48"
Sep 18 20:09:50.220040 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:50.220000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:50.336599 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:50.336567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:09:51.795095 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:09:51.795054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:10.126257 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:10.126224    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:38.272727 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:38.272687    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:45.076769 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:45.076735    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:58.820360 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:58.820326    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-659975bbf5-vhnrs"]
Sep 18 20:10:58.820819 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:58.820511    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-659975bbf5-vhnrs" podUID="7a0c8f85-c704-434a-ad7d-2e36790e4fa5" containerName="authorino" containerID="cri-o://2be0340f730a046901045d19cbf7e581bc6f69ee539238df42bdc98a0fcf74dc" gracePeriod=30
Sep 18 20:10:59.062979 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.062955    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-659975bbf5-vhnrs"
Sep 18 20:10:59.206073 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.205989    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lv8z\" (UniqueName: \"kubernetes.io/projected/7a0c8f85-c704-434a-ad7d-2e36790e4fa5-kube-api-access-6lv8z\") pod \"7a0c8f85-c704-434a-ad7d-2e36790e4fa5\" (UID: \"7a0c8f85-c704-434a-ad7d-2e36790e4fa5\") "
Sep 18 20:10:59.206073 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.206055    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a0c8f85-c704-434a-ad7d-2e36790e4fa5-tls-cert\") pod \"7a0c8f85-c704-434a-ad7d-2e36790e4fa5\" (UID: \"7a0c8f85-c704-434a-ad7d-2e36790e4fa5\") "
Sep 18 20:10:59.208164 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.208136    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0c8f85-c704-434a-ad7d-2e36790e4fa5-kube-api-access-6lv8z" (OuterVolumeSpecName: "kube-api-access-6lv8z") pod "7a0c8f85-c704-434a-ad7d-2e36790e4fa5" (UID: "7a0c8f85-c704-434a-ad7d-2e36790e4fa5"). InnerVolumeSpecName "kube-api-access-6lv8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:10:59.215282 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.215258    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0c8f85-c704-434a-ad7d-2e36790e4fa5-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7a0c8f85-c704-434a-ad7d-2e36790e4fa5" (UID: "7a0c8f85-c704-434a-ad7d-2e36790e4fa5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:10:59.306883 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.306848    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a0c8f85-c704-434a-ad7d-2e36790e4fa5-tls-cert\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:10:59.306883 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.306878    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lv8z\" (UniqueName: \"kubernetes.io/projected/7a0c8f85-c704-434a-ad7d-2e36790e4fa5-kube-api-access-6lv8z\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:10:59.626989 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.626955    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:10:59.818299 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.818268    2578 generic.go:356] "Generic (PLEG): container finished" podID="7a0c8f85-c704-434a-ad7d-2e36790e4fa5" containerID="2be0340f730a046901045d19cbf7e581bc6f69ee539238df42bdc98a0fcf74dc" exitCode=0
Sep 18 20:10:59.818461 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.818310    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-659975bbf5-vhnrs"
Sep 18 20:10:59.818461 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.818353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-659975bbf5-vhnrs" event={"ID":"7a0c8f85-c704-434a-ad7d-2e36790e4fa5","Type":"ContainerDied","Data":"2be0340f730a046901045d19cbf7e581bc6f69ee539238df42bdc98a0fcf74dc"}
Sep 18 20:10:59.818461 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.818392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-659975bbf5-vhnrs" event={"ID":"7a0c8f85-c704-434a-ad7d-2e36790e4fa5","Type":"ContainerDied","Data":"a7fd964aee2dfa3bf4bdae06b2a3cb8dfb2a8a4cf18e102d0742669550ba3cca"}
Sep 18 20:10:59.818461 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.818408    2578 scope.go:117] "RemoveContainer" containerID="2be0340f730a046901045d19cbf7e581bc6f69ee539238df42bdc98a0fcf74dc"
Sep 18 20:10:59.827265 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.827079    2578 scope.go:117] "RemoveContainer" containerID="2be0340f730a046901045d19cbf7e581bc6f69ee539238df42bdc98a0fcf74dc"
Sep 18 20:10:59.827504 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:10:59.827397    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be0340f730a046901045d19cbf7e581bc6f69ee539238df42bdc98a0fcf74dc\": container with ID starting with 2be0340f730a046901045d19cbf7e581bc6f69ee539238df42bdc98a0fcf74dc not found: ID does not exist" containerID="2be0340f730a046901045d19cbf7e581bc6f69ee539238df42bdc98a0fcf74dc"
Sep 18 20:10:59.827504 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.827425    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be0340f730a046901045d19cbf7e581bc6f69ee539238df42bdc98a0fcf74dc"} err="failed to get container status \"2be0340f730a046901045d19cbf7e581bc6f69ee539238df42bdc98a0fcf74dc\": rpc error: code = NotFound desc = could not find container \"2be0340f730a046901045d19cbf7e581bc6f69ee539238df42bdc98a0fcf74dc\": container with ID starting with 2be0340f730a046901045d19cbf7e581bc6f69ee539238df42bdc98a0fcf74dc not found: ID does not exist"
Sep 18 20:10:59.945143 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.945061    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-659975bbf5-vhnrs"]
Sep 18 20:10:59.951404 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:10:59.947126    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-659975bbf5-vhnrs"]
Sep 18 20:11:00.114069 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:00.114022    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:00.317652 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:00.317619    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a0c8f85-c704-434a-ad7d-2e36790e4fa5" path="/var/lib/kubelet/pods/7a0c8f85-c704-434a-ad7d-2e36790e4fa5/volumes"
Sep 18 20:11:01.080802 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:01.080762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:01.716640 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:01.716594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:18.072569 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:18.072538    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:18.374381 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:18.374294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:18.686574 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:18.686454    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:18.951594 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:18.951519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:19.232288 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:19.232254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:19.416403 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:19.416366    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:19.840684 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:19.840645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:20.270155 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:20.270112    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:20.427069 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:20.427041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:22.124411 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:22.124337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:22.285601 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:22.285564    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:23.163570 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:23.163534    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:23.448701 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:23.448620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:23.882117 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:23.882082    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:25.015355 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.015323    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"]
Sep 18 20:11:25.015813 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.015610    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a0c8f85-c704-434a-ad7d-2e36790e4fa5" containerName="authorino"
Sep 18 20:11:25.015813 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.015620    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a0c8f85-c704-434a-ad7d-2e36790e4fa5" containerName="authorino"
Sep 18 20:11:25.015813 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.015684    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a0c8f85-c704-434a-ad7d-2e36790e4fa5" containerName="authorino"
Sep 18 20:11:25.020198 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.020176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.024585 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.024565    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-5284dc-serving-cert\""
Sep 18 20:11:25.024758 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.024735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 20:11:25.025310 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.025290    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8kcjh\""
Sep 18 20:11:25.039292 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.039266    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"]
Sep 18 20:11:25.121091 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.121054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f03d8a5-81b3-42d9-b68f-0e7771c04658-metrics-certs\") pod \"maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.121270 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.121103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47jjj\" (UniqueName: \"kubernetes.io/projected/9f03d8a5-81b3-42d9-b68f-0e7771c04658-kube-api-access-47jjj\") pod \"maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.121270 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.121128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f03d8a5-81b3-42d9-b68f-0e7771c04658-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.121270 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.121163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f03d8a5-81b3-42d9-b68f-0e7771c04658-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.124746 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.124717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:25.221844 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.221807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f03d8a5-81b3-42d9-b68f-0e7771c04658-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.221844 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.221847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f03d8a5-81b3-42d9-b68f-0e7771c04658-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.222095 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.221889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f03d8a5-81b3-42d9-b68f-0e7771c04658-metrics-certs\") pod \"maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.222095 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.221921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47jjj\" (UniqueName: \"kubernetes.io/projected/9f03d8a5-81b3-42d9-b68f-0e7771c04658-kube-api-access-47jjj\") pod \"maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.222603 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.222578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f03d8a5-81b3-42d9-b68f-0e7771c04658-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.224380 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.224357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f03d8a5-81b3-42d9-b68f-0e7771c04658-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.224515 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.224498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f03d8a5-81b3-42d9-b68f-0e7771c04658-metrics-certs\") pod \"maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.232618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.232597    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47jjj\" (UniqueName: \"kubernetes.io/projected/9f03d8a5-81b3-42d9-b68f-0e7771c04658-kube-api-access-47jjj\") pod \"maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.302088 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.302021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:25.330031 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.330002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:25.616545 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.616472    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"]
Sep 18 20:11:25.619280 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:11:25.619249    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f03d8a5_81b3_42d9_b68f_0e7771c04658.slice/crio-7765318cf1539f70950848e94f128da8f9912c617f7306fb95572d25430960e2 WatchSource:0}: Error finding container 7765318cf1539f70950848e94f128da8f9912c617f7306fb95572d25430960e2: Status 404 returned error can't find the container with id 7765318cf1539f70950848e94f128da8f9912c617f7306fb95572d25430960e2
Sep 18 20:11:25.935774 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:25.935685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2" event={"ID":"9f03d8a5-81b3-42d9-b68f-0e7771c04658","Type":"ContainerStarted","Data":"7765318cf1539f70950848e94f128da8f9912c617f7306fb95572d25430960e2"}
Sep 18 20:11:26.120334 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.120305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:26.310025 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.309965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:26.424683 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.424643    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"]
Sep 18 20:11:26.428243 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.428207    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"
Sep 18 20:11:26.431718 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.431686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970fbd29-7b59-452e-99e2-4d351fca4188-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756\" (UID: \"970fbd29-7b59-452e-99e2-4d351fca4188\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"
Sep 18 20:11:26.431857 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.431749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tsc2\" (UniqueName: \"kubernetes.io/projected/970fbd29-7b59-452e-99e2-4d351fca4188-kube-api-access-2tsc2\") pod \"payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756\" (UID: \"970fbd29-7b59-452e-99e2-4d351fca4188\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"
Sep 18 20:11:26.442130 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.442069    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-5284dc-dockercfg-g5v72\""
Sep 18 20:11:26.442130 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.442108    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-5284dc\""
Sep 18 20:11:26.480342 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.480311    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"]
Sep 18 20:11:26.532326 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.532287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tsc2\" (UniqueName: \"kubernetes.io/projected/970fbd29-7b59-452e-99e2-4d351fca4188-kube-api-access-2tsc2\") pod \"payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756\" (UID: \"970fbd29-7b59-452e-99e2-4d351fca4188\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"
Sep 18 20:11:26.532510 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.532422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970fbd29-7b59-452e-99e2-4d351fca4188-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756\" (UID: \"970fbd29-7b59-452e-99e2-4d351fca4188\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"
Sep 18 20:11:26.533358 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.533330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970fbd29-7b59-452e-99e2-4d351fca4188-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756\" (UID: \"970fbd29-7b59-452e-99e2-4d351fca4188\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"
Sep 18 20:11:26.584310 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.584230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tsc2\" (UniqueName: \"kubernetes.io/projected/970fbd29-7b59-452e-99e2-4d351fca4188-kube-api-access-2tsc2\") pod \"payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756\" (UID: \"970fbd29-7b59-452e-99e2-4d351fca4188\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"
Sep 18 20:11:26.743011 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:26.742978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"
Sep 18 20:11:27.069946 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:27.069902    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"]
Sep 18 20:11:27.249907 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:27.249856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:27.517772 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:27.517730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:28.014652 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:28.014613    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:28.152697 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:11:28.152553    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970fbd29_7b59_452e_99e2_4d351fca4188.slice/crio-1ff2b996a191e30efb9cc8c12c0e1babb0f79335c339c03cbe08d4649065dc4c WatchSource:0}: Error finding container 1ff2b996a191e30efb9cc8c12c0e1babb0f79335c339c03cbe08d4649065dc4c: Status 404 returned error can't find the container with id 1ff2b996a191e30efb9cc8c12c0e1babb0f79335c339c03cbe08d4649065dc4c
Sep 18 20:11:28.329171 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:28.329130    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:28.886795 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:28.886681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:28.953998 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:28.953965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2" event={"ID":"9f03d8a5-81b3-42d9-b68f-0e7771c04658","Type":"ContainerStarted","Data":"9c8004e4c31ad4959386ea6560f319794f6d797aa9e36d44a5f7c87453953071"}
Sep 18 20:11:28.954172 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:28.954090    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:28.955639 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:28.955610    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756" event={"ID":"970fbd29-7b59-452e-99e2-4d351fca4188","Type":"ContainerStarted","Data":"1ff2b996a191e30efb9cc8c12c0e1babb0f79335c339c03cbe08d4649065dc4c"}
Sep 18 20:11:29.143959 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.143839    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:29.389759 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.389710    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2" podStartSLOduration=2.8001120459999997 podStartE2EDuration="5.389693315s" totalImagesPullingTime="2.589581269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:29.033383748 +0000 UTC m=+890.871847700" watchObservedRunningTime="2026-09-18 20:11:29.389693315 +0000 UTC m=+891.228157266"
Sep 18 20:11:29.391306 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.391277    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"]
Sep 18 20:11:29.394696 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.394659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"
Sep 18 20:11:29.413995 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.413970    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-4b633d-dockercfg-qrb5x\""
Sep 18 20:11:29.414104 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.414010    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-4b633d\""
Sep 18 20:11:29.424464 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.424432    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"]
Sep 18 20:11:29.460135 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.460105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpbrh\" (UniqueName: \"kubernetes.io/projected/e81f22d6-c572-4c5f-920f-661c9a82f430-kube-api-access-wpbrh\") pod \"payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84\" (UID: \"e81f22d6-c572-4c5f-920f-661c9a82f430\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"
Sep 18 20:11:29.460289 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.460268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81f22d6-c572-4c5f-920f-661c9a82f430-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84\" (UID: \"e81f22d6-c572-4c5f-920f-661c9a82f430\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"
Sep 18 20:11:29.561132 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.561093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81f22d6-c572-4c5f-920f-661c9a82f430-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84\" (UID: \"e81f22d6-c572-4c5f-920f-661c9a82f430\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"
Sep 18 20:11:29.561328 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.561143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbrh\" (UniqueName: \"kubernetes.io/projected/e81f22d6-c572-4c5f-920f-661c9a82f430-kube-api-access-wpbrh\") pod \"payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84\" (UID: \"e81f22d6-c572-4c5f-920f-661c9a82f430\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"
Sep 18 20:11:29.561783 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.561761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81f22d6-c572-4c5f-920f-661c9a82f430-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84\" (UID: \"e81f22d6-c572-4c5f-920f-661c9a82f430\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"
Sep 18 20:11:29.582537 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.582509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbrh\" (UniqueName: \"kubernetes.io/projected/e81f22d6-c572-4c5f-920f-661c9a82f430-kube-api-access-wpbrh\") pod \"payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84\" (UID: \"e81f22d6-c572-4c5f-920f-661c9a82f430\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"
Sep 18 20:11:29.705607 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.705509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"
Sep 18 20:11:29.963950 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.963892    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756" event={"ID":"970fbd29-7b59-452e-99e2-4d351fca4188","Type":"ContainerStarted","Data":"01f8883cee42036c9b486b9614f50e144250580dafc3a1dc8e4ad64b5974ccc1"}
Sep 18 20:11:29.964246 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.964222    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"
Sep 18 20:11:29.991868 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:29.991819    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756" podStartSLOduration=2.755137649 podStartE2EDuration="3.991801857s" totalImagesPullingTime="1.236664208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:29.990497864 +0000 UTC m=+891.828961816" watchObservedRunningTime="2026-09-18 20:11:29.991801857 +0000 UTC m=+891.830265811"
Sep 18 20:11:30.021819 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.021789    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"]
Sep 18 20:11:30.026024 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:11:30.025991    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81f22d6_c572_4c5f_920f_661c9a82f430.slice/crio-7746d8d5caf32509b7e118a426c5c9ff43875806a9e6e2a9c5634233493e35b9 WatchSource:0}: Error finding container 7746d8d5caf32509b7e118a426c5c9ff43875806a9e6e2a9c5634233493e35b9: Status 404 returned error can't find the container with id 7746d8d5caf32509b7e118a426c5c9ff43875806a9e6e2a9c5634233493e35b9
Sep 18 20:11:30.301435 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.301351    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"]
Sep 18 20:11:30.304540 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.304524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:30.324118 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.324086    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3a8052-serving-cert\""
Sep 18 20:11:30.367558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.367523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-metrics-certs\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:30.367675 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.367581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5067c12c-2149-4dcc-a372-9956709543f9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:30.367675 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.367655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:30.367758 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.367685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgbl6\" (UniqueName: \"kubernetes.io/projected/5067c12c-2149-4dcc-a372-9956709543f9-kube-api-access-lgbl6\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:30.424517 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.424487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:30.468186 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.468148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-metrics-certs\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:30.468346 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.468200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5067c12c-2149-4dcc-a372-9956709543f9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:30.468346 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.468256    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:30.468346 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.468281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgbl6\" (UniqueName: \"kubernetes.io/projected/5067c12c-2149-4dcc-a372-9956709543f9-kube-api-access-lgbl6\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:30.468517 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:11:30.468451    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-serving-cert: secret "maas-api-e2e-inf-a-3a8052-serving-cert" not found
Sep 18 20:11:30.468572 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:11:30.468518    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-maas-api-tls podName:5067c12c-2149-4dcc-a372-9956709543f9 nodeName:}" failed. No retries permitted until 2026-09-18 20:11:30.968499133 +0000 UTC m=+892.806963082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-maas-api-tls") pod "maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx" (UID: "5067c12c-2149-4dcc-a372-9956709543f9") : secret "maas-api-e2e-inf-a-3a8052-serving-cert" not found
Sep 18 20:11:30.469195 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.469175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5067c12c-2149-4dcc-a372-9956709543f9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:30.471027 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.471002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-metrics-certs\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:30.484342 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.484319    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgbl6\" (UniqueName: \"kubernetes.io/projected/5067c12c-2149-4dcc-a372-9956709543f9-kube-api-access-lgbl6\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:30.575407 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.575330    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"]
Sep 18 20:11:30.699778 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.699750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:30.969984 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.969953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84" event={"ID":"e81f22d6-c572-4c5f-920f-661c9a82f430","Type":"ContainerStarted","Data":"4c05b4528baf1e105fc2e983f7634c71e0156a67b1a6f31318741fb410dcf28d"}
Sep 18 20:11:30.970149 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.970023    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"
Sep 18 20:11:30.970149 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.970045    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84" event={"ID":"e81f22d6-c572-4c5f-920f-661c9a82f430","Type":"ContainerStarted","Data":"7746d8d5caf32509b7e118a426c5c9ff43875806a9e6e2a9c5634233493e35b9"}
Sep 18 20:11:30.972707 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:30.972686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:30.972823 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:11:30.972782    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-serving-cert: secret "maas-api-e2e-inf-a-3a8052-serving-cert" not found
Sep 18 20:11:30.972858 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:11:30.972828    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-maas-api-tls podName:5067c12c-2149-4dcc-a372-9956709543f9 nodeName:}" failed. No retries permitted until 2026-09-18 20:11:31.972814456 +0000 UTC m=+893.811278387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-maas-api-tls") pod "maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx" (UID: "5067c12c-2149-4dcc-a372-9956709543f9") : secret "maas-api-e2e-inf-a-3a8052-serving-cert" not found
Sep 18 20:11:31.001968 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.001909    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84" podStartSLOduration=2.001895751 podStartE2EDuration="2.001895751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:31.000121009 +0000 UTC m=+892.838584981" watchObservedRunningTime="2026-09-18 20:11:31.001895751 +0000 UTC m=+892.840359729"
Sep 18 20:11:31.120447 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.120409    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:31.503775 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.503740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:31.690681 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.690646    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:31.764287 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.764211    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"]
Sep 18 20:11:31.768671 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.768654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"
Sep 18 20:11:31.771324 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.771302    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3a8052\""
Sep 18 20:11:31.791176 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.791149    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"]
Sep 18 20:11:31.859106 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.859070    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:31.881287 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.881256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8qw9\" (UniqueName: \"kubernetes.io/projected/f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70-kube-api-access-k8qw9\") pod \"payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww\" (UID: \"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"
Sep 18 20:11:31.881441 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.881303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww\" (UID: \"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"
Sep 18 20:11:31.982657 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.982626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8qw9\" (UniqueName: \"kubernetes.io/projected/f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70-kube-api-access-k8qw9\") pod \"payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww\" (UID: \"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"
Sep 18 20:11:31.982837 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.982671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww\" (UID: \"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"
Sep 18 20:11:31.982837 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.982791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:31.983497 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.983478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww\" (UID: \"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"
Sep 18 20:11:31.985123 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.985107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-maas-api-tls\") pod \"maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:31.996106 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:31.996071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8qw9\" (UniqueName: \"kubernetes.io/projected/f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70-kube-api-access-k8qw9\") pod \"payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww\" (UID: \"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"
Sep 18 20:11:32.079649 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:32.079586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"
Sep 18 20:11:32.117006 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:32.116718    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:32.359285 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:32.359202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:32.374852 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:32.374828    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"]
Sep 18 20:11:32.376209 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:11:32.376180    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cfaa30_ace3_4d7b_bbd6_3613b7b73f70.slice/crio-ee66711d6affd9ac005ccfa4c165d11909c8b16f28baf8efc191ed692320cb74 WatchSource:0}: Error finding container ee66711d6affd9ac005ccfa4c165d11909c8b16f28baf8efc191ed692320cb74: Status 404 returned error can't find the container with id ee66711d6affd9ac005ccfa4c165d11909c8b16f28baf8efc191ed692320cb74
Sep 18 20:11:32.430030 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:32.429999    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"]
Sep 18 20:11:32.431964 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:11:32.431927    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5067c12c_2149_4dcc_a372_9956709543f9.slice/crio-5cc48d622995f3bff7679181299f7fc5d5762378ce24248fe30b28c3e344d44b WatchSource:0}: Error finding container 5cc48d622995f3bff7679181299f7fc5d5762378ce24248fe30b28c3e344d44b: Status 404 returned error can't find the container with id 5cc48d622995f3bff7679181299f7fc5d5762378ce24248fe30b28c3e344d44b
Sep 18 20:11:32.980709 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:32.980677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx" event={"ID":"5067c12c-2149-4dcc-a372-9956709543f9","Type":"ContainerStarted","Data":"89232538f6a2fedd38fbbaa147a773a256b36493047fc5b8a4afe60880a8c7de"}
Sep 18 20:11:32.981189 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:32.980869    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:32.981189 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:32.980945    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx" event={"ID":"5067c12c-2149-4dcc-a372-9956709543f9","Type":"ContainerStarted","Data":"5cc48d622995f3bff7679181299f7fc5d5762378ce24248fe30b28c3e344d44b"}
Sep 18 20:11:32.982406 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:32.982388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww" event={"ID":"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70","Type":"ContainerStarted","Data":"f40cf8e65487143507449755544386f4f5707b797bdd80ce727a7ce97dc74539"}
Sep 18 20:11:32.982474 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:32.982424    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww" event={"ID":"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70","Type":"ContainerStarted","Data":"ee66711d6affd9ac005ccfa4c165d11909c8b16f28baf8efc191ed692320cb74"}
Sep 18 20:11:32.982474 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:32.982462    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"
Sep 18 20:11:33.021547 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:33.021434    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx" podStartSLOduration=2.684166248 podStartE2EDuration="3.021420615s" totalImagesPullingTime="337.254367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:33.020614348 +0000 UTC m=+894.859078300" watchObservedRunningTime="2026-09-18 20:11:33.021420615 +0000 UTC m=+894.859884566"
Sep 18 20:11:33.058556 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:33.058508    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww" podStartSLOduration=2.058493591 podStartE2EDuration="2.058493591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:33.056227429 +0000 UTC m=+894.894691382" watchObservedRunningTime="2026-09-18 20:11:33.058493591 +0000 UTC m=+894.896957542"
Sep 18 20:11:34.025899 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:34.025857    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:34.970040 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:34.970006    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:11:35.329090 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:35.329002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:35.569021 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:35.568980    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:35.960065 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:35.960026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:36.267008 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:36.266977    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:36.511428 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:36.511384    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:36.794163 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:36.794127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:37.395741 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:37.395710    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:37.686152 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:37.686068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:38.002059 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:38.002018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:38.298633 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:38.298539    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:38.619804 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:38.619714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:38.888168 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:38.888067    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:38.992133 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:38.992105    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:11:39.386609 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:39.386574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:39.745615 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:39.745575    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:40.456032 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:40.455993    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:40.972794 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:40.972765    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"
Sep 18 20:11:41.222196 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:41.222158    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:41.680350 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:41.680315    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:41.976139 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:41.976112    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"
Sep 18 20:11:42.065581 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.065546    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:42.221588 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.221547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:42.564052 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.564015    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"]
Sep 18 20:11:42.568767 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.568751    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"
Sep 18 20:11:42.571917 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.571893    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-d0d77c-dockercfg-65dld\""
Sep 18 20:11:42.572275 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.572241    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:42.572394 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.572306    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-d0d77c\""
Sep 18 20:11:42.588778 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.588755    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"]
Sep 18 20:11:42.629956 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.629913    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"]
Sep 18 20:11:42.633171 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.633155    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"
Sep 18 20:11:42.661445 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.661401    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"]
Sep 18 20:11:42.680728 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.680694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e11b92fd-5ff0-4474-9fc8-8bc63e38fa24-plugins-config-volume\") pod \"payload-processing-e2e-autores-d0d77c-649b889959-bkvfh\" (UID: \"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24\") " pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"
Sep 18 20:11:42.681114 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.680767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956411fb-81d9-4c44-82c7-2825ef4f2aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn\" (UID: \"956411fb-81d9-4c44-82c7-2825ef4f2aba\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"
Sep 18 20:11:42.681114 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.680870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk96q\" (UniqueName: \"kubernetes.io/projected/e11b92fd-5ff0-4474-9fc8-8bc63e38fa24-kube-api-access-tk96q\") pod \"payload-processing-e2e-autores-d0d77c-649b889959-bkvfh\" (UID: \"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24\") " pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"
Sep 18 20:11:42.681114 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.680918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j879c\" (UniqueName: \"kubernetes.io/projected/956411fb-81d9-4c44-82c7-2825ef4f2aba-kube-api-access-j879c\") pod \"payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn\" (UID: \"956411fb-81d9-4c44-82c7-2825ef4f2aba\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"
Sep 18 20:11:42.725134 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.725106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:42.782292 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.782262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk96q\" (UniqueName: \"kubernetes.io/projected/e11b92fd-5ff0-4474-9fc8-8bc63e38fa24-kube-api-access-tk96q\") pod \"payload-processing-e2e-autores-d0d77c-649b889959-bkvfh\" (UID: \"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24\") " pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"
Sep 18 20:11:42.782429 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.782298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j879c\" (UniqueName: \"kubernetes.io/projected/956411fb-81d9-4c44-82c7-2825ef4f2aba-kube-api-access-j879c\") pod \"payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn\" (UID: \"956411fb-81d9-4c44-82c7-2825ef4f2aba\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"
Sep 18 20:11:42.782429 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.782337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e11b92fd-5ff0-4474-9fc8-8bc63e38fa24-plugins-config-volume\") pod \"payload-processing-e2e-autores-d0d77c-649b889959-bkvfh\" (UID: \"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24\") " pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"
Sep 18 20:11:42.782429 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.782370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956411fb-81d9-4c44-82c7-2825ef4f2aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn\" (UID: \"956411fb-81d9-4c44-82c7-2825ef4f2aba\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"
Sep 18 20:11:42.783089 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.783071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e11b92fd-5ff0-4474-9fc8-8bc63e38fa24-plugins-config-volume\") pod \"payload-processing-e2e-autores-d0d77c-649b889959-bkvfh\" (UID: \"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24\") " pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"
Sep 18 20:11:42.783135 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.783103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956411fb-81d9-4c44-82c7-2825ef4f2aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn\" (UID: \"956411fb-81d9-4c44-82c7-2825ef4f2aba\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"
Sep 18 20:11:42.800515 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.800492    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk96q\" (UniqueName: \"kubernetes.io/projected/e11b92fd-5ff0-4474-9fc8-8bc63e38fa24-kube-api-access-tk96q\") pod \"payload-processing-e2e-autores-d0d77c-649b889959-bkvfh\" (UID: \"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24\") " pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"
Sep 18 20:11:42.800617 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.800588    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j879c\" (UniqueName: \"kubernetes.io/projected/956411fb-81d9-4c44-82c7-2825ef4f2aba-kube-api-access-j879c\") pod \"payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn\" (UID: \"956411fb-81d9-4c44-82c7-2825ef4f2aba\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"
Sep 18 20:11:42.874759 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.874685    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:42.878547 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.878509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"
Sep 18 20:11:42.943758 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:42.943576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"
Sep 18 20:11:43.048686 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:43.048651    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:43.178048 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:43.178024    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"]
Sep 18 20:11:43.180534 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:11:43.180508    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956411fb_81d9_4c44_82c7_2825ef4f2aba.slice/crio-93563270c1b65819709dd5140d17c946cbf585d083cd9b6e2ea581d485e68780 WatchSource:0}: Error finding container 93563270c1b65819709dd5140d17c946cbf585d083cd9b6e2ea581d485e68780: Status 404 returned error can't find the container with id 93563270c1b65819709dd5140d17c946cbf585d083cd9b6e2ea581d485e68780
Sep 18 20:11:43.244074 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:43.244048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"]
Sep 18 20:11:43.247292 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:11:43.247269    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11b92fd_5ff0_4474_9fc8_8bc63e38fa24.slice/crio-43bff496d1e3a486d14c5f658ed5a00a44e37f5d483fa2a414a3885e97b415ff WatchSource:0}: Error finding container 43bff496d1e3a486d14c5f658ed5a00a44e37f5d483fa2a414a3885e97b415ff: Status 404 returned error can't find the container with id 43bff496d1e3a486d14c5f658ed5a00a44e37f5d483fa2a414a3885e97b415ff
Sep 18 20:11:43.531820 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:43.531780    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:43.748178 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:43.748141    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:43.989162 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:43.989132    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"
Sep 18 20:11:44.041664 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:44.041628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh" event={"ID":"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24","Type":"ContainerStarted","Data":"e803de6a14ef37acc742a5b7b8ee0c94ebf749d2ce4c8e74568a7edd647998f6"}
Sep 18 20:11:44.041886 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:44.041690    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh" event={"ID":"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24","Type":"ContainerStarted","Data":"43bff496d1e3a486d14c5f658ed5a00a44e37f5d483fa2a414a3885e97b415ff"}
Sep 18 20:11:44.041886 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:44.041722    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"
Sep 18 20:11:44.043273 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:44.043248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn" event={"ID":"956411fb-81d9-4c44-82c7-2825ef4f2aba","Type":"ContainerStarted","Data":"4d6f4b014aa0edd79d5f1ad4bfaf71c2d6120afa8255330ce0bede098e65babe"}
Sep 18 20:11:44.043370 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:44.043292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn" event={"ID":"956411fb-81d9-4c44-82c7-2825ef4f2aba","Type":"ContainerStarted","Data":"93563270c1b65819709dd5140d17c946cbf585d083cd9b6e2ea581d485e68780"}
Sep 18 20:11:44.043423 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:44.043373    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"
Sep 18 20:11:44.067817 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:44.067749    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh" podStartSLOduration=2.067732396 podStartE2EDuration="2.067732396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:44.065177524 +0000 UTC m=+905.903641478" watchObservedRunningTime="2026-09-18 20:11:44.067732396 +0000 UTC m=+905.906196351"
Sep 18 20:11:44.115038 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:44.114990    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn" podStartSLOduration=2.11497557 podStartE2EDuration="2.11497557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:44.111286345 +0000 UTC m=+905.949750297" watchObservedRunningTime="2026-09-18 20:11:44.11497557 +0000 UTC m=+905.953439521"
Sep 18 20:11:45.491723 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:45.491683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:47.051097 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:47.051052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:47.313818 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:47.313724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:47.422552 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:47.422517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:47.805176 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:47.805126    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:48.271628 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.271596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:48.626561 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.626478    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"]
Sep 18 20:11:48.630656 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.630636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:48.633713 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.633684    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0a7e3fae-serving-cert\""
Sep 18 20:11:48.641386 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.641360    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"]
Sep 18 20:11:48.671432 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.671408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:48.735876 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.735840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2mp\" (UniqueName: \"kubernetes.io/projected/48d198d9-164e-4712-a98b-ad2158f620d7-kube-api-access-xj2mp\") pod \"maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:48.736047 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.735913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d198d9-164e-4712-a98b-ad2158f620d7-openshift-service-ca\") pod \"maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:48.736047 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.735998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-maas-api-tls\") pod \"maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:48.736047 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.736031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-metrics-certs\") pod \"maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:48.781119 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.781076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:48.837410 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.837377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-maas-api-tls\") pod \"maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:48.837545 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.837413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-metrics-certs\") pod \"maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:48.837545 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:11:48.837511    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-serving-cert: secret "maas-api-e2e-ait-0a7e3fae-serving-cert" not found
Sep 18 20:11:48.837545 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.837536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2mp\" (UniqueName: \"kubernetes.io/projected/48d198d9-164e-4712-a98b-ad2158f620d7-kube-api-access-xj2mp\") pod \"maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:48.837666 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:11:48.837600    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-maas-api-tls podName:48d198d9-164e-4712-a98b-ad2158f620d7 nodeName:}" failed. No retries permitted until 2026-09-18 20:11:49.337584596 +0000 UTC m=+911.176048527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-maas-api-tls") pod "maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8" (UID: "48d198d9-164e-4712-a98b-ad2158f620d7") : secret "maas-api-e2e-ait-0a7e3fae-serving-cert" not found
Sep 18 20:11:48.837666 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.837631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d198d9-164e-4712-a98b-ad2158f620d7-openshift-service-ca\") pod \"maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:48.838220 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.838200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d198d9-164e-4712-a98b-ad2158f620d7-openshift-service-ca\") pod \"maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:48.839876 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.839855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-metrics-certs\") pod \"maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:48.855652 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:48.855625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2mp\" (UniqueName: \"kubernetes.io/projected/48d198d9-164e-4712-a98b-ad2158f620d7-kube-api-access-xj2mp\") pod \"maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:49.009793 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.009760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:49.343065 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.342984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-maas-api-tls\") pod \"maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:49.345759 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.345733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-maas-api-tls\") pod \"maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:49.406651 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.406617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:49.522684 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.522653    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:49.542567 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.542535    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:49.698189 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.698155    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"]
Sep 18 20:11:49.702880 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.702859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"
Sep 18 20:11:49.706068 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.706045    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0a7e3fae\""
Sep 18 20:11:49.706339 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.706320    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0a7e3fae-dockercfg-s5kl8\""
Sep 18 20:11:49.717801 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.717774    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"]
Sep 18 20:11:49.821228 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.821199    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"]
Sep 18 20:11:49.822567 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:11:49.822542    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d198d9_164e_4712_a98b_ad2158f620d7.slice/crio-05bda181ef5ef507b2e3564827d79b51a7ee972bd433b4e419030936d7ddfaf0 WatchSource:0}: Error finding container 05bda181ef5ef507b2e3564827d79b51a7ee972bd433b4e419030936d7ddfaf0: Status 404 returned error can't find the container with id 05bda181ef5ef507b2e3564827d79b51a7ee972bd433b4e419030936d7ddfaf0
Sep 18 20:11:49.846747 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.846669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66gnw\" (UniqueName: \"kubernetes.io/projected/e3e2b1bf-de79-47cc-b64d-f34f48cac939-kube-api-access-66gnw\") pod \"payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64\" (UID: \"e3e2b1bf-de79-47cc-b64d-f34f48cac939\") " pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"
Sep 18 20:11:49.846747 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.846706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e2b1bf-de79-47cc-b64d-f34f48cac939-plugins-config-volume\") pod \"payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64\" (UID: \"e3e2b1bf-de79-47cc-b64d-f34f48cac939\") " pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"
Sep 18 20:11:49.948146 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.948109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66gnw\" (UniqueName: \"kubernetes.io/projected/e3e2b1bf-de79-47cc-b64d-f34f48cac939-kube-api-access-66gnw\") pod \"payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64\" (UID: \"e3e2b1bf-de79-47cc-b64d-f34f48cac939\") " pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"
Sep 18 20:11:49.948362 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.948167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e2b1bf-de79-47cc-b64d-f34f48cac939-plugins-config-volume\") pod \"payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64\" (UID: \"e3e2b1bf-de79-47cc-b64d-f34f48cac939\") " pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"
Sep 18 20:11:49.948867 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.948847    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e2b1bf-de79-47cc-b64d-f34f48cac939-plugins-config-volume\") pod \"payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64\" (UID: \"e3e2b1bf-de79-47cc-b64d-f34f48cac939\") " pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"
Sep 18 20:11:49.959389 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:49.959359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66gnw\" (UniqueName: \"kubernetes.io/projected/e3e2b1bf-de79-47cc-b64d-f34f48cac939-kube-api-access-66gnw\") pod \"payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64\" (UID: \"e3e2b1bf-de79-47cc-b64d-f34f48cac939\") " pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"
Sep 18 20:11:50.016627 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:50.016520    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"
Sep 18 20:11:50.057983 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:50.057925    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:50.081617 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:50.081581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8" event={"ID":"48d198d9-164e-4712-a98b-ad2158f620d7","Type":"ContainerStarted","Data":"05bda181ef5ef507b2e3564827d79b51a7ee972bd433b4e419030936d7ddfaf0"}
Sep 18 20:11:50.295138 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:50.295112    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"]
Sep 18 20:11:50.296899 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:11:50.296873    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e2b1bf_de79_47cc_b64d_f34f48cac939.slice/crio-720a71d7e5d9502649633b16d459069f1448bafb54baba1e88f2401707917938 WatchSource:0}: Error finding container 720a71d7e5d9502649633b16d459069f1448bafb54baba1e88f2401707917938: Status 404 returned error can't find the container with id 720a71d7e5d9502649633b16d459069f1448bafb54baba1e88f2401707917938
Sep 18 20:11:51.091065 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:51.090431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8" event={"ID":"48d198d9-164e-4712-a98b-ad2158f620d7","Type":"ContainerStarted","Data":"bfa9d66f9f4f571cc9ef17937443454a3aec3dd8f8c61e2f8a0aa937a63e6d93"}
Sep 18 20:11:51.091065 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:51.090987    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:51.092732 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:51.092708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64" event={"ID":"e3e2b1bf-de79-47cc-b64d-f34f48cac939","Type":"ContainerStarted","Data":"940df137a5cf9f0ba1fb06df8027b3ede6863363be72beea433c74f1fe632666"}
Sep 18 20:11:51.092891 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:51.092751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64" event={"ID":"e3e2b1bf-de79-47cc-b64d-f34f48cac939","Type":"ContainerStarted","Data":"720a71d7e5d9502649633b16d459069f1448bafb54baba1e88f2401707917938"}
Sep 18 20:11:51.092891 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:51.092779    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"
Sep 18 20:11:51.122207 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:51.122160    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8" podStartSLOduration=2.832806739 podStartE2EDuration="3.122145308s" totalImagesPullingTime="289.338569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:51.120431167 +0000 UTC m=+912.958895120" watchObservedRunningTime="2026-09-18 20:11:51.122145308 +0000 UTC m=+912.960609259"
Sep 18 20:11:51.129962 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:51.129920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:51.149323 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:51.149282    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64" podStartSLOduration=2.149268214 podStartE2EDuration="2.149268214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:51.147581871 +0000 UTC m=+912.986045823" watchObservedRunningTime="2026-09-18 20:11:51.149268214 +0000 UTC m=+912.987732166"
Sep 18 20:11:51.228411 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:51.228371    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:51.972756 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:51.972715    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:52.164301 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:52.164241    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:52.299558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:52.299473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:52.422331 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:52.422293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:52.524352 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:52.524314    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:53.637792 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:53.637756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:54.931876 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:54.931835    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:55.050410 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:55.050379    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"
Sep 18 20:11:55.050577 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:55.050453    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"
Sep 18 20:11:55.967204 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:55.967152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:56.067794 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:56.067750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:56.168956 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:56.168917    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:56.428466 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:56.428389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:56.741202 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:56.741170    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:56.883682 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:56.883652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:56.970205 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:56.970169    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"]
Sep 18 20:11:56.973354 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:56.973337    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"
Sep 18 20:11:56.975804 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:56.975777    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c34fad\""
Sep 18 20:11:56.976075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:56.976060    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c34fad-dockercfg-r7f42\""
Sep 18 20:11:56.986603 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:56.986579    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"]
Sep 18 20:11:57.104255 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:57.104176    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:11:57.114686 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:57.114657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4hll\" (UniqueName: \"kubernetes.io/projected/390d9f6b-73ee-480b-b514-b4a48a09a31e-kube-api-access-h4hll\") pod \"payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp\" (UID: \"390d9f6b-73ee-480b-b514-b4a48a09a31e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"
Sep 18 20:11:57.114836 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:57.114716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390d9f6b-73ee-480b-b514-b4a48a09a31e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp\" (UID: \"390d9f6b-73ee-480b-b514-b4a48a09a31e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"
Sep 18 20:11:57.215868 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:57.215838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390d9f6b-73ee-480b-b514-b4a48a09a31e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp\" (UID: \"390d9f6b-73ee-480b-b514-b4a48a09a31e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"
Sep 18 20:11:57.216055 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:57.215990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4hll\" (UniqueName: \"kubernetes.io/projected/390d9f6b-73ee-480b-b514-b4a48a09a31e-kube-api-access-h4hll\") pod \"payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp\" (UID: \"390d9f6b-73ee-480b-b514-b4a48a09a31e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"
Sep 18 20:11:57.216706 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:57.216680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390d9f6b-73ee-480b-b514-b4a48a09a31e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp\" (UID: \"390d9f6b-73ee-480b-b514-b4a48a09a31e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"
Sep 18 20:11:57.225902 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:57.225876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4hll\" (UniqueName: \"kubernetes.io/projected/390d9f6b-73ee-480b-b514-b4a48a09a31e-kube-api-access-h4hll\") pod \"payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp\" (UID: \"390d9f6b-73ee-480b-b514-b4a48a09a31e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"
Sep 18 20:11:57.283956 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:57.283902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"
Sep 18 20:11:57.373722 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:57.373650    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:57.560108 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:57.560081    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"]
Sep 18 20:11:57.562475 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:11:57.562450    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390d9f6b_73ee_480b_b514_b4a48a09a31e.slice/crio-d3ebfa3f439ae5f37afe46a05c4dae139a8c7c7376653583f57daa64054372b0 WatchSource:0}: Error finding container d3ebfa3f439ae5f37afe46a05c4dae139a8c7c7376653583f57daa64054372b0: Status 404 returned error can't find the container with id d3ebfa3f439ae5f37afe46a05c4dae139a8c7c7376653583f57daa64054372b0
Sep 18 20:11:58.131357 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:58.131321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp" event={"ID":"390d9f6b-73ee-480b-b514-b4a48a09a31e","Type":"ContainerStarted","Data":"d801ab16f1f31fdb06232fa092565a8ebd32c738913295757698078c3ee0a4d1"}
Sep 18 20:11:58.131751 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:58.131375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp" event={"ID":"390d9f6b-73ee-480b-b514-b4a48a09a31e","Type":"ContainerStarted","Data":"d3ebfa3f439ae5f37afe46a05c4dae139a8c7c7376653583f57daa64054372b0"}
Sep 18 20:11:58.131751 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:58.131469    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"
Sep 18 20:11:58.631545 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:58.631511    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:59.597631 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:59.597574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:59.789019 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:59.788985    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:11:59.905294 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:11:59.905210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:00.517097 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:00.517057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:01.482213 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:01.482182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:02.100436 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:02.100405    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"
Sep 18 20:12:02.122162 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:02.122108    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp" podStartSLOduration=6.122092153 podStartE2EDuration="6.122092153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:11:58.203747025 +0000 UTC m=+920.042210977" watchObservedRunningTime="2026-09-18 20:12:02.122092153 +0000 UTC m=+923.960556141"
Sep 18 20:12:02.378410 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:02.378321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:03.397913 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:03.397874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:04.789108 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:04.789069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:06.305729 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:06.305693    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:07.660875 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:07.660835    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:08.559197 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:08.559163    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:09.138436 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:09.138403    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"
Sep 18 20:12:09.694560 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:09.694507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:10.540145 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:10.540106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:11.644701 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:11.644670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:12.574427 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:12.574388    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:13.530949 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:13.530894    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:14.553114 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:14.553072    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:15.364621 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:15.364574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:15.953996 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:15.953954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:16.240146 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:16.240118    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:17.642870 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:17.642775    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:18.700390 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:18.700354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:22.712954 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:22.712904    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"]
Sep 18 20:12:22.713442 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:22.713134    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8" podUID="48d198d9-164e-4712-a98b-ad2158f620d7" containerName="maas-api" containerID="cri-o://bfa9d66f9f4f571cc9ef17937443454a3aec3dd8f8c61e2f8a0aa937a63e6d93" gracePeriod=30
Sep 18 20:12:22.742459 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:22.742420    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:22.784595 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:22.784558    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"]
Sep 18 20:12:22.784846 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:22.784819    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64" podUID="e3e2b1bf-de79-47cc-b64d-f34f48cac939" containerName="payload-processing" containerID="cri-o://940df137a5cf9f0ba1fb06df8027b3ede6863363be72beea433c74f1fe632666" gracePeriod=30
Sep 18 20:12:22.855431 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:22.855400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:22.961350 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:22.961316    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:23.037411 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.037388    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:12:23.060296 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.060276    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"
Sep 18 20:12:23.121339 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.121302    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:23.138611 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.138586    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-maas-api-tls\") pod \"48d198d9-164e-4712-a98b-ad2158f620d7\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") "
Sep 18 20:12:23.138731 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.138637    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66gnw\" (UniqueName: \"kubernetes.io/projected/e3e2b1bf-de79-47cc-b64d-f34f48cac939-kube-api-access-66gnw\") pod \"e3e2b1bf-de79-47cc-b64d-f34f48cac939\" (UID: \"e3e2b1bf-de79-47cc-b64d-f34f48cac939\") "
Sep 18 20:12:23.138731 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.138684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-metrics-certs\") pod \"48d198d9-164e-4712-a98b-ad2158f620d7\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") "
Sep 18 20:12:23.138731 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.138710    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e2b1bf-de79-47cc-b64d-f34f48cac939-plugins-config-volume\") pod \"e3e2b1bf-de79-47cc-b64d-f34f48cac939\" (UID: \"e3e2b1bf-de79-47cc-b64d-f34f48cac939\") "
Sep 18 20:12:23.138853 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.138732    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d198d9-164e-4712-a98b-ad2158f620d7-openshift-service-ca\") pod \"48d198d9-164e-4712-a98b-ad2158f620d7\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") "
Sep 18 20:12:23.138853 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.138799    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj2mp\" (UniqueName: \"kubernetes.io/projected/48d198d9-164e-4712-a98b-ad2158f620d7-kube-api-access-xj2mp\") pod \"48d198d9-164e-4712-a98b-ad2158f620d7\" (UID: \"48d198d9-164e-4712-a98b-ad2158f620d7\") "
Sep 18 20:12:23.139597 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.139550    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d198d9-164e-4712-a98b-ad2158f620d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "48d198d9-164e-4712-a98b-ad2158f620d7" (UID: "48d198d9-164e-4712-a98b-ad2158f620d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:12:23.139716 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.139634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e2b1bf-de79-47cc-b64d-f34f48cac939-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e3e2b1bf-de79-47cc-b64d-f34f48cac939" (UID: "e3e2b1bf-de79-47cc-b64d-f34f48cac939"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:12:23.141924 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.141895    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "48d198d9-164e-4712-a98b-ad2158f620d7" (UID: "48d198d9-164e-4712-a98b-ad2158f620d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:12:23.143050 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.143024    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e2b1bf-de79-47cc-b64d-f34f48cac939-kube-api-access-66gnw" (OuterVolumeSpecName: "kube-api-access-66gnw") pod "e3e2b1bf-de79-47cc-b64d-f34f48cac939" (UID: "e3e2b1bf-de79-47cc-b64d-f34f48cac939"). InnerVolumeSpecName "kube-api-access-66gnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:12:23.146620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.146590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "48d198d9-164e-4712-a98b-ad2158f620d7" (UID: "48d198d9-164e-4712-a98b-ad2158f620d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:12:23.147444 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.147423    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d198d9-164e-4712-a98b-ad2158f620d7-kube-api-access-xj2mp" (OuterVolumeSpecName: "kube-api-access-xj2mp") pod "48d198d9-164e-4712-a98b-ad2158f620d7" (UID: "48d198d9-164e-4712-a98b-ad2158f620d7"). InnerVolumeSpecName "kube-api-access-xj2mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:12:23.223278 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.223239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:23.239612 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.239546    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:12:23.239612 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.239589    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e2b1bf-de79-47cc-b64d-f34f48cac939-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:12:23.239612 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.239601    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d198d9-164e-4712-a98b-ad2158f620d7-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:12:23.239612 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.239611    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2mp\" (UniqueName: \"kubernetes.io/projected/48d198d9-164e-4712-a98b-ad2158f620d7-kube-api-access-xj2mp\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:12:23.239859 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.239620    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48d198d9-164e-4712-a98b-ad2158f620d7-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:12:23.239859 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.239629    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66gnw\" (UniqueName: \"kubernetes.io/projected/e3e2b1bf-de79-47cc-b64d-f34f48cac939-kube-api-access-66gnw\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:12:23.258041 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.258012    2578 generic.go:356] "Generic (PLEG): container finished" podID="48d198d9-164e-4712-a98b-ad2158f620d7" containerID="bfa9d66f9f4f571cc9ef17937443454a3aec3dd8f8c61e2f8a0aa937a63e6d93" exitCode=0
Sep 18 20:12:23.258179 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.258072    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"
Sep 18 20:12:23.258179 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.258102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8" event={"ID":"48d198d9-164e-4712-a98b-ad2158f620d7","Type":"ContainerDied","Data":"bfa9d66f9f4f571cc9ef17937443454a3aec3dd8f8c61e2f8a0aa937a63e6d93"}
Sep 18 20:12:23.258302 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.258187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8" event={"ID":"48d198d9-164e-4712-a98b-ad2158f620d7","Type":"ContainerDied","Data":"05bda181ef5ef507b2e3564827d79b51a7ee972bd433b4e419030936d7ddfaf0"}
Sep 18 20:12:23.258302 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.258216    2578 scope.go:117] "RemoveContainer" containerID="bfa9d66f9f4f571cc9ef17937443454a3aec3dd8f8c61e2f8a0aa937a63e6d93"
Sep 18 20:12:23.259622 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.259603    2578 generic.go:356] "Generic (PLEG): container finished" podID="e3e2b1bf-de79-47cc-b64d-f34f48cac939" containerID="940df137a5cf9f0ba1fb06df8027b3ede6863363be72beea433c74f1fe632666" exitCode=0
Sep 18 20:12:23.259743 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.259658    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"
Sep 18 20:12:23.259743 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.259665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64" event={"ID":"e3e2b1bf-de79-47cc-b64d-f34f48cac939","Type":"ContainerDied","Data":"940df137a5cf9f0ba1fb06df8027b3ede6863363be72beea433c74f1fe632666"}
Sep 18 20:12:23.259743 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.259708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64" event={"ID":"e3e2b1bf-de79-47cc-b64d-f34f48cac939","Type":"ContainerDied","Data":"720a71d7e5d9502649633b16d459069f1448bafb54baba1e88f2401707917938"}
Sep 18 20:12:23.268470 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.268455    2578 scope.go:117] "RemoveContainer" containerID="bfa9d66f9f4f571cc9ef17937443454a3aec3dd8f8c61e2f8a0aa937a63e6d93"
Sep 18 20:12:23.268741 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:12:23.268723    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa9d66f9f4f571cc9ef17937443454a3aec3dd8f8c61e2f8a0aa937a63e6d93\": container with ID starting with bfa9d66f9f4f571cc9ef17937443454a3aec3dd8f8c61e2f8a0aa937a63e6d93 not found: ID does not exist" containerID="bfa9d66f9f4f571cc9ef17937443454a3aec3dd8f8c61e2f8a0aa937a63e6d93"
Sep 18 20:12:23.268806 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.268748    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa9d66f9f4f571cc9ef17937443454a3aec3dd8f8c61e2f8a0aa937a63e6d93"} err="failed to get container status \"bfa9d66f9f4f571cc9ef17937443454a3aec3dd8f8c61e2f8a0aa937a63e6d93\": rpc error: code = NotFound desc = could not find container \"bfa9d66f9f4f571cc9ef17937443454a3aec3dd8f8c61e2f8a0aa937a63e6d93\": container with ID starting with bfa9d66f9f4f571cc9ef17937443454a3aec3dd8f8c61e2f8a0aa937a63e6d93 not found: ID does not exist"
Sep 18 20:12:23.268806 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.268769    2578 scope.go:117] "RemoveContainer" containerID="940df137a5cf9f0ba1fb06df8027b3ede6863363be72beea433c74f1fe632666"
Sep 18 20:12:23.276552 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.276538    2578 scope.go:117] "RemoveContainer" containerID="940df137a5cf9f0ba1fb06df8027b3ede6863363be72beea433c74f1fe632666"
Sep 18 20:12:23.276799 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:12:23.276782    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940df137a5cf9f0ba1fb06df8027b3ede6863363be72beea433c74f1fe632666\": container with ID starting with 940df137a5cf9f0ba1fb06df8027b3ede6863363be72beea433c74f1fe632666 not found: ID does not exist" containerID="940df137a5cf9f0ba1fb06df8027b3ede6863363be72beea433c74f1fe632666"
Sep 18 20:12:23.276859 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.276804    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940df137a5cf9f0ba1fb06df8027b3ede6863363be72beea433c74f1fe632666"} err="failed to get container status \"940df137a5cf9f0ba1fb06df8027b3ede6863363be72beea433c74f1fe632666\": rpc error: code = NotFound desc = could not find container \"940df137a5cf9f0ba1fb06df8027b3ede6863363be72beea433c74f1fe632666\": container with ID starting with 940df137a5cf9f0ba1fb06df8027b3ede6863363be72beea433c74f1fe632666 not found: ID does not exist"
Sep 18 20:12:23.286363 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.286339    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"]
Sep 18 20:12:23.298480 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.298460    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0a7e3fae-6847c7cc57-p8jc8"]
Sep 18 20:12:23.312459 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.312437    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"]
Sep 18 20:12:23.317047 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.317024    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a7e3fae-776d9784f5-zhk64"]
Sep 18 20:12:23.326066 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.326037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:23.416497 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.416458    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:23.778143 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.778108    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:23.898213 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.898178    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:23.999153 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:23.999075    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:24.315311 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:24.315209    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:24.318169 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:24.318144    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48d198d9-164e-4712-a98b-ad2158f620d7" path="/var/lib/kubelet/pods/48d198d9-164e-4712-a98b-ad2158f620d7/volumes"
Sep 18 20:12:24.318579 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:24.318563    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e2b1bf-de79-47cc-b64d-f34f48cac939" path="/var/lib/kubelet/pods/e3e2b1bf-de79-47cc-b64d-f34f48cac939/volumes"
Sep 18 20:12:24.565169 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:24.565132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:24.725163 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:24.725130    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:25.926929 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:25.926896    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:26.836080 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:26.836044    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:26.936925 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:26.936893    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:27.857071 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:27.857031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:29.010547 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:29.010515    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:29.902319 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:29.902285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:30.857042 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.857005    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"]
Sep 18 20:12:30.857412 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.857359    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e2b1bf-de79-47cc-b64d-f34f48cac939" containerName="payload-processing"
Sep 18 20:12:30.857412 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.857370    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e2b1bf-de79-47cc-b64d-f34f48cac939" containerName="payload-processing"
Sep 18 20:12:30.857412 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.857382    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d198d9-164e-4712-a98b-ad2158f620d7" containerName="maas-api"
Sep 18 20:12:30.857412 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.857387    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d198d9-164e-4712-a98b-ad2158f620d7" containerName="maas-api"
Sep 18 20:12:30.857545 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.857445    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d198d9-164e-4712-a98b-ad2158f620d7" containerName="maas-api"
Sep 18 20:12:30.857545 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.857454    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e2b1bf-de79-47cc-b64d-f34f48cac939" containerName="payload-processing"
Sep 18 20:12:30.860642 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.860624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:30.865647 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.865625    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7bed33dc-serving-cert\""
Sep 18 20:12:30.875473 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.875448    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"]
Sep 18 20:12:30.893295 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.893267    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:30.901241 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.901216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b56532f-26e1-4e18-863c-7179cd0b3eef-metrics-certs\") pod \"maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:30.901355 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.901252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b56532f-26e1-4e18-863c-7179cd0b3eef-maas-api-tls\") pod \"maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:30.901355 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.901313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwwhf\" (UniqueName: \"kubernetes.io/projected/1b56532f-26e1-4e18-863c-7179cd0b3eef-kube-api-access-wwwhf\") pod \"maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:30.901355 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:30.901351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b56532f-26e1-4e18-863c-7179cd0b3eef-openshift-service-ca\") pod \"maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:31.002489 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.002453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b56532f-26e1-4e18-863c-7179cd0b3eef-maas-api-tls\") pod \"maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:31.002670 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.002513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwwhf\" (UniqueName: \"kubernetes.io/projected/1b56532f-26e1-4e18-863c-7179cd0b3eef-kube-api-access-wwwhf\") pod \"maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:31.002670 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.002560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b56532f-26e1-4e18-863c-7179cd0b3eef-openshift-service-ca\") pod \"maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:31.002670 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.002594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b56532f-26e1-4e18-863c-7179cd0b3eef-metrics-certs\") pod \"maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:31.003284 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.003260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b56532f-26e1-4e18-863c-7179cd0b3eef-openshift-service-ca\") pod \"maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:31.005015 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.004985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b56532f-26e1-4e18-863c-7179cd0b3eef-maas-api-tls\") pod \"maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:31.005119 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.005052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b56532f-26e1-4e18-863c-7179cd0b3eef-metrics-certs\") pod \"maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:31.025489 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.025465    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:31.035524 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.035504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwwhf\" (UniqueName: \"kubernetes.io/projected/1b56532f-26e1-4e18-863c-7179cd0b3eef-kube-api-access-wwwhf\") pod \"maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:31.168624 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.168529    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:31.170452 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.170431    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:31.464501 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.464474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"]
Sep 18 20:12:31.470641 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:12:31.470604    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b56532f_26e1_4e18_863c_7179cd0b3eef.slice/crio-b1d5ccf22656114f52501129e8b780977d1fa01fb6af2135bf81727c9e076101 WatchSource:0}: Error finding container b1d5ccf22656114f52501129e8b780977d1fa01fb6af2135bf81727c9e076101: Status 404 returned error can't find the container with id b1d5ccf22656114f52501129e8b780977d1fa01fb6af2135bf81727c9e076101
Sep 18 20:12:31.693202 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.693164    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:31.786729 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.786637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:31.826125 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.826088    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"]
Sep 18 20:12:31.830905 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.830881    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"
Sep 18 20:12:31.835037 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.835015    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7bed33dc\""
Sep 18 20:12:31.846473 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.846446    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"]
Sep 18 20:12:31.912134 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.912097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef9796a-4b0e-4608-831a-dfdc6a899b12-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph\" (UID: \"7ef9796a-4b0e-4608-831a-dfdc6a899b12\") " pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"
Sep 18 20:12:31.912502 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:31.912143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9nds\" (UniqueName: \"kubernetes.io/projected/7ef9796a-4b0e-4608-831a-dfdc6a899b12-kube-api-access-d9nds\") pod \"payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph\" (UID: \"7ef9796a-4b0e-4608-831a-dfdc6a899b12\") " pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"
Sep 18 20:12:32.013186 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:32.013163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef9796a-4b0e-4608-831a-dfdc6a899b12-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph\" (UID: \"7ef9796a-4b0e-4608-831a-dfdc6a899b12\") " pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"
Sep 18 20:12:32.013295 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:32.013202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9nds\" (UniqueName: \"kubernetes.io/projected/7ef9796a-4b0e-4608-831a-dfdc6a899b12-kube-api-access-d9nds\") pod \"payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph\" (UID: \"7ef9796a-4b0e-4608-831a-dfdc6a899b12\") " pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"
Sep 18 20:12:32.013853 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:32.013833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef9796a-4b0e-4608-831a-dfdc6a899b12-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph\" (UID: \"7ef9796a-4b0e-4608-831a-dfdc6a899b12\") " pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"
Sep 18 20:12:32.023346 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:32.023324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9nds\" (UniqueName: \"kubernetes.io/projected/7ef9796a-4b0e-4608-831a-dfdc6a899b12-kube-api-access-d9nds\") pod \"payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph\" (UID: \"7ef9796a-4b0e-4608-831a-dfdc6a899b12\") " pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"
Sep 18 20:12:32.141244 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:32.141210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"
Sep 18 20:12:32.307462 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:32.307372    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q" event={"ID":"1b56532f-26e1-4e18-863c-7179cd0b3eef","Type":"ContainerStarted","Data":"3310fff82804cefffd84c0b64ae016de8f8203440310b3c0d23f7c29b98af9a5"}
Sep 18 20:12:32.307462 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:32.307422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q" event={"ID":"1b56532f-26e1-4e18-863c-7179cd0b3eef","Type":"ContainerStarted","Data":"b1d5ccf22656114f52501129e8b780977d1fa01fb6af2135bf81727c9e076101"}
Sep 18 20:12:32.307689 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:32.307475    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:32.330351 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:32.330302    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q" podStartSLOduration=1.828321511 podStartE2EDuration="2.330287004s" totalImagesPullingTime="501.965493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:32.32774842 +0000 UTC m=+954.166212372" watchObservedRunningTime="2026-09-18 20:12:32.330287004 +0000 UTC m=+954.168750955"
Sep 18 20:12:32.376777 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:32.376748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:32.414411 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:32.414388    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"]
Sep 18 20:12:32.416276 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:12:32.416253    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef9796a_4b0e_4608_831a_dfdc6a899b12.slice/crio-56b0f0758056ffe7168ca5be08e152bf1a12e2f8a820761ddd6d2e33dc2540c5 WatchSource:0}: Error finding container 56b0f0758056ffe7168ca5be08e152bf1a12e2f8a820761ddd6d2e33dc2540c5: Status 404 returned error can't find the container with id 56b0f0758056ffe7168ca5be08e152bf1a12e2f8a820761ddd6d2e33dc2540c5
Sep 18 20:12:33.280119 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:33.280058    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:33.314216 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:33.314181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph" event={"ID":"7ef9796a-4b0e-4608-831a-dfdc6a899b12","Type":"ContainerStarted","Data":"d06cae6666d1ec1875e4f05c459194593927f59039eef77cc10e5a783aaf2000"}
Sep 18 20:12:33.314392 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:33.314268    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph" event={"ID":"7ef9796a-4b0e-4608-831a-dfdc6a899b12","Type":"ContainerStarted","Data":"56b0f0758056ffe7168ca5be08e152bf1a12e2f8a820761ddd6d2e33dc2540c5"}
Sep 18 20:12:33.314466 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:33.314443    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"
Sep 18 20:12:33.335322 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:33.335268    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph" podStartSLOduration=2.335253146 podStartE2EDuration="2.335253146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:33.33402564 +0000 UTC m=+955.172489594" watchObservedRunningTime="2026-09-18 20:12:33.335253146 +0000 UTC m=+955.173717112"
Sep 18 20:12:34.129450 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:34.129411    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:38.320279 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:38.320253    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:12:44.320668 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:44.320637    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"
Sep 18 20:12:56.472815 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:56.472741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:57.433918 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:57.433885    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:58.348637 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:58.348608    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:59.164200 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:59.164163    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:12:59.953577 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:12:59.953534    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:00.763882 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:00.763853    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:01.604325 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:01.604295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:02.437413 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:02.437381    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:04.114620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:04.114585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:08.021073 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:08.021041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:08.772574 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:08.772542    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"]
Sep 18 20:13:08.772870 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:08.772844    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q" podUID="1b56532f-26e1-4e18-863c-7179cd0b3eef" containerName="maas-api" containerID="cri-o://3310fff82804cefffd84c0b64ae016de8f8203440310b3c0d23f7c29b98af9a5" gracePeriod=30
Sep 18 20:13:08.812036 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:08.812000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:08.845650 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:08.845617    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"]
Sep 18 20:13:08.845925 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:08.845876    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph" podUID="7ef9796a-4b0e-4608-831a-dfdc6a899b12" containerName="payload-processing" containerID="cri-o://d06cae6666d1ec1875e4f05c459194593927f59039eef77cc10e5a783aaf2000" gracePeriod=30
Sep 18 20:13:08.908502 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:08.908474    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:09.009191 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.009153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:09.077479 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.077380    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:09.093122 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.093099    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:13:09.129327 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.129302    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b56532f-26e1-4e18-863c-7179cd0b3eef-maas-api-tls\") pod \"1b56532f-26e1-4e18-863c-7179cd0b3eef\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") "
Sep 18 20:13:09.129470 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.129448    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b56532f-26e1-4e18-863c-7179cd0b3eef-metrics-certs\") pod \"1b56532f-26e1-4e18-863c-7179cd0b3eef\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") "
Sep 18 20:13:09.129543 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.129484    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwwhf\" (UniqueName: \"kubernetes.io/projected/1b56532f-26e1-4e18-863c-7179cd0b3eef-kube-api-access-wwwhf\") pod \"1b56532f-26e1-4e18-863c-7179cd0b3eef\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") "
Sep 18 20:13:09.129601 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.129559    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b56532f-26e1-4e18-863c-7179cd0b3eef-openshift-service-ca\") pod \"1b56532f-26e1-4e18-863c-7179cd0b3eef\" (UID: \"1b56532f-26e1-4e18-863c-7179cd0b3eef\") "
Sep 18 20:13:09.130065 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.130036    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b56532f-26e1-4e18-863c-7179cd0b3eef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b56532f-26e1-4e18-863c-7179cd0b3eef" (UID: "1b56532f-26e1-4e18-863c-7179cd0b3eef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:13:09.131610 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.131588    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b56532f-26e1-4e18-863c-7179cd0b3eef-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1b56532f-26e1-4e18-863c-7179cd0b3eef" (UID: "1b56532f-26e1-4e18-863c-7179cd0b3eef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:13:09.131699 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.131635    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b56532f-26e1-4e18-863c-7179cd0b3eef-kube-api-access-wwwhf" (OuterVolumeSpecName: "kube-api-access-wwwhf") pod "1b56532f-26e1-4e18-863c-7179cd0b3eef" (UID: "1b56532f-26e1-4e18-863c-7179cd0b3eef"). InnerVolumeSpecName "kube-api-access-wwwhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:13:09.131798 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.131785    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b56532f-26e1-4e18-863c-7179cd0b3eef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b56532f-26e1-4e18-863c-7179cd0b3eef" (UID: "1b56532f-26e1-4e18-863c-7179cd0b3eef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:13:09.132648 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.132623    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"
Sep 18 20:13:09.228348 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.228312    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:09.230318 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.230296    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9nds\" (UniqueName: \"kubernetes.io/projected/7ef9796a-4b0e-4608-831a-dfdc6a899b12-kube-api-access-d9nds\") pod \"7ef9796a-4b0e-4608-831a-dfdc6a899b12\" (UID: \"7ef9796a-4b0e-4608-831a-dfdc6a899b12\") "
Sep 18 20:13:09.230438 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.230340    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef9796a-4b0e-4608-831a-dfdc6a899b12-plugins-config-volume\") pod \"7ef9796a-4b0e-4608-831a-dfdc6a899b12\" (UID: \"7ef9796a-4b0e-4608-831a-dfdc6a899b12\") "
Sep 18 20:13:09.230592 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.230576    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b56532f-26e1-4e18-863c-7179cd0b3eef-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:13:09.230651 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.230597    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b56532f-26e1-4e18-863c-7179cd0b3eef-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:13:09.230651 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.230611    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b56532f-26e1-4e18-863c-7179cd0b3eef-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:13:09.230651 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.230625    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwwhf\" (UniqueName: \"kubernetes.io/projected/1b56532f-26e1-4e18-863c-7179cd0b3eef-kube-api-access-wwwhf\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:13:09.230750 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.230702    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef9796a-4b0e-4608-831a-dfdc6a899b12-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ef9796a-4b0e-4608-831a-dfdc6a899b12" (UID: "7ef9796a-4b0e-4608-831a-dfdc6a899b12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:13:09.232432 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.232413    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef9796a-4b0e-4608-831a-dfdc6a899b12-kube-api-access-d9nds" (OuterVolumeSpecName: "kube-api-access-d9nds") pod "7ef9796a-4b0e-4608-831a-dfdc6a899b12" (UID: "7ef9796a-4b0e-4608-831a-dfdc6a899b12"). InnerVolumeSpecName "kube-api-access-d9nds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:13:09.331740 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.331664    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9nds\" (UniqueName: \"kubernetes.io/projected/7ef9796a-4b0e-4608-831a-dfdc6a899b12-kube-api-access-d9nds\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:13:09.331740 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.331692    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef9796a-4b0e-4608-831a-dfdc6a899b12-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:13:09.490670 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.490635    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b56532f-26e1-4e18-863c-7179cd0b3eef" containerID="3310fff82804cefffd84c0b64ae016de8f8203440310b3c0d23f7c29b98af9a5" exitCode=0
Sep 18 20:13:09.490670 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.490672    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"
Sep 18 20:13:09.490910 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.490716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q" event={"ID":"1b56532f-26e1-4e18-863c-7179cd0b3eef","Type":"ContainerDied","Data":"3310fff82804cefffd84c0b64ae016de8f8203440310b3c0d23f7c29b98af9a5"}
Sep 18 20:13:09.490910 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.490771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q" event={"ID":"1b56532f-26e1-4e18-863c-7179cd0b3eef","Type":"ContainerDied","Data":"b1d5ccf22656114f52501129e8b780977d1fa01fb6af2135bf81727c9e076101"}
Sep 18 20:13:09.490910 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.490799    2578 scope.go:117] "RemoveContainer" containerID="3310fff82804cefffd84c0b64ae016de8f8203440310b3c0d23f7c29b98af9a5"
Sep 18 20:13:09.492064 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.492042    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ef9796a-4b0e-4608-831a-dfdc6a899b12" containerID="d06cae6666d1ec1875e4f05c459194593927f59039eef77cc10e5a783aaf2000" exitCode=0
Sep 18 20:13:09.492175 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.492098    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"
Sep 18 20:13:09.492175 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.492132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph" event={"ID":"7ef9796a-4b0e-4608-831a-dfdc6a899b12","Type":"ContainerDied","Data":"d06cae6666d1ec1875e4f05c459194593927f59039eef77cc10e5a783aaf2000"}
Sep 18 20:13:09.492175 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.492169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph" event={"ID":"7ef9796a-4b0e-4608-831a-dfdc6a899b12","Type":"ContainerDied","Data":"56b0f0758056ffe7168ca5be08e152bf1a12e2f8a820761ddd6d2e33dc2540c5"}
Sep 18 20:13:09.501097 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.501073    2578 scope.go:117] "RemoveContainer" containerID="3310fff82804cefffd84c0b64ae016de8f8203440310b3c0d23f7c29b98af9a5"
Sep 18 20:13:09.501465 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:13:09.501428    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3310fff82804cefffd84c0b64ae016de8f8203440310b3c0d23f7c29b98af9a5\": container with ID starting with 3310fff82804cefffd84c0b64ae016de8f8203440310b3c0d23f7c29b98af9a5 not found: ID does not exist" containerID="3310fff82804cefffd84c0b64ae016de8f8203440310b3c0d23f7c29b98af9a5"
Sep 18 20:13:09.501573 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.501465    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3310fff82804cefffd84c0b64ae016de8f8203440310b3c0d23f7c29b98af9a5"} err="failed to get container status \"3310fff82804cefffd84c0b64ae016de8f8203440310b3c0d23f7c29b98af9a5\": rpc error: code = NotFound desc = could not find container \"3310fff82804cefffd84c0b64ae016de8f8203440310b3c0d23f7c29b98af9a5\": container with ID starting with 3310fff82804cefffd84c0b64ae016de8f8203440310b3c0d23f7c29b98af9a5 not found: ID does not exist"
Sep 18 20:13:09.501573 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.501489    2578 scope.go:117] "RemoveContainer" containerID="d06cae6666d1ec1875e4f05c459194593927f59039eef77cc10e5a783aaf2000"
Sep 18 20:13:09.510547 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.510524    2578 scope.go:117] "RemoveContainer" containerID="d06cae6666d1ec1875e4f05c459194593927f59039eef77cc10e5a783aaf2000"
Sep 18 20:13:09.510823 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:13:09.510795    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06cae6666d1ec1875e4f05c459194593927f59039eef77cc10e5a783aaf2000\": container with ID starting with d06cae6666d1ec1875e4f05c459194593927f59039eef77cc10e5a783aaf2000 not found: ID does not exist" containerID="d06cae6666d1ec1875e4f05c459194593927f59039eef77cc10e5a783aaf2000"
Sep 18 20:13:09.510927 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.510833    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06cae6666d1ec1875e4f05c459194593927f59039eef77cc10e5a783aaf2000"} err="failed to get container status \"d06cae6666d1ec1875e4f05c459194593927f59039eef77cc10e5a783aaf2000\": rpc error: code = NotFound desc = could not find container \"d06cae6666d1ec1875e4f05c459194593927f59039eef77cc10e5a783aaf2000\": container with ID starting with d06cae6666d1ec1875e4f05c459194593927f59039eef77cc10e5a783aaf2000 not found: ID does not exist"
Sep 18 20:13:09.511031 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.510927    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:09.522648 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.522622    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"]
Sep 18 20:13:09.524616 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.524596    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7bed33dc-7574cd898-8mvph"]
Sep 18 20:13:09.535032 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.535007    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"]
Sep 18 20:13:09.542085 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.542066    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7bed33dc-784c58bc59-gjv7q"]
Sep 18 20:13:09.700269 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:09.700174    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:10.024841 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:10.024807    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:10.318456 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:10.318379    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b56532f-26e1-4e18-863c-7179cd0b3eef" path="/var/lib/kubelet/pods/1b56532f-26e1-4e18-863c-7179cd0b3eef/volumes"
Sep 18 20:13:10.318808 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:10.318727    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ef9796a-4b0e-4608-831a-dfdc6a899b12" path="/var/lib/kubelet/pods/7ef9796a-4b0e-4608-831a-dfdc6a899b12/volumes"
Sep 18 20:13:10.737465 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:10.737433    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:10.839297 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:10.839261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:11.642974 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:11.642942    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:12.574095 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:12.574062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:13.031580 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:13.031523    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:13.124037 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:13.123985    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:13.542357 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:13.542322    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:14.375013 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:14.374984    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:15.246320 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.246285    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"]
Sep 18 20:13:15.246648 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.246635    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef9796a-4b0e-4608-831a-dfdc6a899b12" containerName="payload-processing"
Sep 18 20:13:15.246648 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.246649    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef9796a-4b0e-4608-831a-dfdc6a899b12" containerName="payload-processing"
Sep 18 20:13:15.246729 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.246661    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b56532f-26e1-4e18-863c-7179cd0b3eef" containerName="maas-api"
Sep 18 20:13:15.246729 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.246667    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b56532f-26e1-4e18-863c-7179cd0b3eef" containerName="maas-api"
Sep 18 20:13:15.246797 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.246732    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ef9796a-4b0e-4608-831a-dfdc6a899b12" containerName="payload-processing"
Sep 18 20:13:15.246797 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.246741    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b56532f-26e1-4e18-863c-7179cd0b3eef" containerName="maas-api"
Sep 18 20:13:15.251286 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.251268    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:15.253503 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.253482    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f524d4ec-serving-cert\""
Sep 18 20:13:15.261292 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.261269    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"]
Sep 18 20:13:15.289918 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.289887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-metrics-certs\") pod \"maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:15.290079 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.289926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-maas-api-tls\") pod \"maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:15.290159 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.290069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ca14f1-53eb-4bae-9610-3258ec585039-openshift-service-ca\") pod \"maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:15.290159 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.290118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkx8j\" (UniqueName: \"kubernetes.io/projected/45ca14f1-53eb-4bae-9610-3258ec585039-kube-api-access-fkx8j\") pod \"maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:15.321728 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.321702    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:15.390716 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.390688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-metrics-certs\") pod \"maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:15.391139 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.390725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-maas-api-tls\") pod \"maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:15.391139 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.390766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ca14f1-53eb-4bae-9610-3258ec585039-openshift-service-ca\") pod \"maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:15.391139 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.390789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkx8j\" (UniqueName: \"kubernetes.io/projected/45ca14f1-53eb-4bae-9610-3258ec585039-kube-api-access-fkx8j\") pod \"maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:15.391139 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:13:15.390910    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-serving-cert: secret "maas-api-e2e-ait-f524d4ec-serving-cert" not found
Sep 18 20:13:15.391139 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:13:15.391019    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-maas-api-tls podName:45ca14f1-53eb-4bae-9610-3258ec585039 nodeName:}" failed. No retries permitted until 2026-09-18 20:13:15.890995604 +0000 UTC m=+997.729459542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-maas-api-tls") pod "maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r" (UID: "45ca14f1-53eb-4bae-9610-3258ec585039") : secret "maas-api-e2e-ait-f524d4ec-serving-cert" not found
Sep 18 20:13:15.391576 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.391552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ca14f1-53eb-4bae-9610-3258ec585039-openshift-service-ca\") pod \"maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:15.393378 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.393359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-metrics-certs\") pod \"maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:15.400529 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.400502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkx8j\" (UniqueName: \"kubernetes.io/projected/45ca14f1-53eb-4bae-9610-3258ec585039-kube-api-access-fkx8j\") pod \"maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:15.419052 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.419023    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:15.644902 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.644810    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:15.895493 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.895394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-maas-api-tls\") pod \"maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:15.897850 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:15.897819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-maas-api-tls\") pod \"maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:16.102747 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.102716    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:16.162070 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.161998    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:16.203778 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.203732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:16.258329 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.258298    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"]
Sep 18 20:13:16.262040 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.262024    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"
Sep 18 20:13:16.264411 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.264386    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f524d4ec-dockercfg-jm6tl\""
Sep 18 20:13:16.264530 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.264386    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f524d4ec\""
Sep 18 20:13:16.275474 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.275452    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"]
Sep 18 20:13:16.299275 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.299239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e9081f6-4035-4074-af23-11bd22ddce24-plugins-config-volume\") pod \"payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r\" (UID: \"1e9081f6-4035-4074-af23-11bd22ddce24\") " pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"
Sep 18 20:13:16.299412 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.299311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7966b\" (UniqueName: \"kubernetes.io/projected/1e9081f6-4035-4074-af23-11bd22ddce24-kube-api-access-7966b\") pod \"payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r\" (UID: \"1e9081f6-4035-4074-af23-11bd22ddce24\") " pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"
Sep 18 20:13:16.400402 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.400363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e9081f6-4035-4074-af23-11bd22ddce24-plugins-config-volume\") pod \"payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r\" (UID: \"1e9081f6-4035-4074-af23-11bd22ddce24\") " pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"
Sep 18 20:13:16.400869 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.400443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7966b\" (UniqueName: \"kubernetes.io/projected/1e9081f6-4035-4074-af23-11bd22ddce24-kube-api-access-7966b\") pod \"payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r\" (UID: \"1e9081f6-4035-4074-af23-11bd22ddce24\") " pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"
Sep 18 20:13:16.401085 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.401067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e9081f6-4035-4074-af23-11bd22ddce24-plugins-config-volume\") pod \"payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r\" (UID: \"1e9081f6-4035-4074-af23-11bd22ddce24\") " pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"
Sep 18 20:13:16.408520 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.408496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7966b\" (UniqueName: \"kubernetes.io/projected/1e9081f6-4035-4074-af23-11bd22ddce24-kube-api-access-7966b\") pod \"payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r\" (UID: \"1e9081f6-4035-4074-af23-11bd22ddce24\") " pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"
Sep 18 20:13:16.436572 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.436541    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"]
Sep 18 20:13:16.439105 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:13:16.439081    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ca14f1_53eb_4bae_9610_3258ec585039.slice/crio-d575bc4cf47e5be160bc1585abf308f1ea3e48d419845a7a788f6e2b20fcfe20 WatchSource:0}: Error finding container d575bc4cf47e5be160bc1585abf308f1ea3e48d419845a7a788f6e2b20fcfe20: Status 404 returned error can't find the container with id d575bc4cf47e5be160bc1585abf308f1ea3e48d419845a7a788f6e2b20fcfe20
Sep 18 20:13:16.527545 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.527511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r" event={"ID":"45ca14f1-53eb-4bae-9610-3258ec585039","Type":"ContainerStarted","Data":"d575bc4cf47e5be160bc1585abf308f1ea3e48d419845a7a788f6e2b20fcfe20"}
Sep 18 20:13:16.572805 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.572772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"
Sep 18 20:13:16.855443 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:16.855410    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"]
Sep 18 20:13:16.856801 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:13:16.856769    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9081f6_4035_4074_af23_11bd22ddce24.slice/crio-7f8e1ebe52221240600451ca2ed6c2f946defb924b372ffbab1e9a1e1d04c6d4 WatchSource:0}: Error finding container 7f8e1ebe52221240600451ca2ed6c2f946defb924b372ffbab1e9a1e1d04c6d4: Status 404 returned error can't find the container with id 7f8e1ebe52221240600451ca2ed6c2f946defb924b372ffbab1e9a1e1d04c6d4
Sep 18 20:13:17.483699 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:17.483658    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:17.535027 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:17.534990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r" event={"ID":"1e9081f6-4035-4074-af23-11bd22ddce24","Type":"ContainerStarted","Data":"788da2bffe508d7e135de8e17c88f91e5d545e6eb4ae22613c3225cf50b8562d"}
Sep 18 20:13:17.535203 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:17.535063    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"
Sep 18 20:13:17.535203 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:17.535090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r" event={"ID":"1e9081f6-4035-4074-af23-11bd22ddce24","Type":"ContainerStarted","Data":"7f8e1ebe52221240600451ca2ed6c2f946defb924b372ffbab1e9a1e1d04c6d4"}
Sep 18 20:13:17.536651 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:17.536627    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r" event={"ID":"45ca14f1-53eb-4bae-9610-3258ec585039","Type":"ContainerStarted","Data":"6c0eb77dce3a7f11f76557f18bbeb28713810b4acb05aaed836791d36dbb9158"}
Sep 18 20:13:17.536772 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:17.536677    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:17.555160 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:17.555119    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r" podStartSLOduration=1.5551057959999999 podStartE2EDuration="1.555105796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:17.552885687 +0000 UTC m=+999.391349639" watchObservedRunningTime="2026-09-18 20:13:17.555105796 +0000 UTC m=+999.393569792"
Sep 18 20:13:17.573008 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:17.572963    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r" podStartSLOduration=2.134764709 podStartE2EDuration="2.572949448s" totalImagesPullingTime="438.184739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:17.572420019 +0000 UTC m=+999.410883971" watchObservedRunningTime="2026-09-18 20:13:17.572949448 +0000 UTC m=+999.411413393"
Sep 18 20:13:18.342527 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:18.342492    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:19.132958 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:19.132894    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:23.547771 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:23.547739    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:28.544107 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:28.544066    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"
Sep 18 20:13:34.741867 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:34.741825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:34.842306 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:34.842273    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:36.982261 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:36.982219    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:39.250423 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:39.250382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:39.354430 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:39.354391    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:40.027669 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:40.027631    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:41.666581 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:41.666543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:42.585675 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:42.585642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:43.376790 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:43.376751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:44.372661 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:44.372633    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:45.247893 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:45.247850    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:46.011519 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:46.011479    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:46.744584 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:46.744550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:47.713539 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:47.713500    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:48.717156 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:48.717123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:49.411705 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:49.411676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:54.215917 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:54.215876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:54.345043 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:54.345009    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:55.073907 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.073875    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"]
Sep 18 20:13:55.074167 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.074143    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r" podUID="45ca14f1-53eb-4bae-9610-3258ec585039" containerName="maas-api" containerID="cri-o://6c0eb77dce3a7f11f76557f18bbeb28713810b4acb05aaed836791d36dbb9158" gracePeriod=30
Sep 18 20:13:55.102117 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.102083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:55.135129 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.135098    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"]
Sep 18 20:13:55.135360 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.135335    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r" podUID="1e9081f6-4035-4074-af23-11bd22ddce24" containerName="payload-processing" containerID="cri-o://788da2bffe508d7e135de8e17c88f91e5d545e6eb4ae22613c3225cf50b8562d" gracePeriod=30
Sep 18 20:13:55.180822 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.180784    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:55.281840 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.281774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:55.382112 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.382083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:55.382422 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.382404    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:55.418892 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.418858    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"
Sep 18 20:13:55.431240 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.431211    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-metrics-certs\") pod \"45ca14f1-53eb-4bae-9610-3258ec585039\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") "
Sep 18 20:13:55.431387 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.431266    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkx8j\" (UniqueName: \"kubernetes.io/projected/45ca14f1-53eb-4bae-9610-3258ec585039-kube-api-access-fkx8j\") pod \"45ca14f1-53eb-4bae-9610-3258ec585039\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") "
Sep 18 20:13:55.431469 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.431388    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-maas-api-tls\") pod \"45ca14f1-53eb-4bae-9610-3258ec585039\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") "
Sep 18 20:13:55.431469 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.431441    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7966b\" (UniqueName: \"kubernetes.io/projected/1e9081f6-4035-4074-af23-11bd22ddce24-kube-api-access-7966b\") pod \"1e9081f6-4035-4074-af23-11bd22ddce24\" (UID: \"1e9081f6-4035-4074-af23-11bd22ddce24\") "
Sep 18 20:13:55.431587 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.431536    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e9081f6-4035-4074-af23-11bd22ddce24-plugins-config-volume\") pod \"1e9081f6-4035-4074-af23-11bd22ddce24\" (UID: \"1e9081f6-4035-4074-af23-11bd22ddce24\") "
Sep 18 20:13:55.431587 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.431563    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ca14f1-53eb-4bae-9610-3258ec585039-openshift-service-ca\") pod \"45ca14f1-53eb-4bae-9610-3258ec585039\" (UID: \"45ca14f1-53eb-4bae-9610-3258ec585039\") "
Sep 18 20:13:55.432434 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.432147    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ca14f1-53eb-4bae-9610-3258ec585039-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45ca14f1-53eb-4bae-9610-3258ec585039" (UID: "45ca14f1-53eb-4bae-9610-3258ec585039"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:13:55.432597 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.432441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9081f6-4035-4074-af23-11bd22ddce24-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e9081f6-4035-4074-af23-11bd22ddce24" (UID: "1e9081f6-4035-4074-af23-11bd22ddce24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:13:55.433699 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.433672    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45ca14f1-53eb-4bae-9610-3258ec585039" (UID: "45ca14f1-53eb-4bae-9610-3258ec585039"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:13:55.433927 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.433910    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "45ca14f1-53eb-4bae-9610-3258ec585039" (UID: "45ca14f1-53eb-4bae-9610-3258ec585039"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:13:55.434000 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.433978    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9081f6-4035-4074-af23-11bd22ddce24-kube-api-access-7966b" (OuterVolumeSpecName: "kube-api-access-7966b") pod "1e9081f6-4035-4074-af23-11bd22ddce24" (UID: "1e9081f6-4035-4074-af23-11bd22ddce24"). InnerVolumeSpecName "kube-api-access-7966b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:13:55.434046 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.434015    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ca14f1-53eb-4bae-9610-3258ec585039-kube-api-access-fkx8j" (OuterVolumeSpecName: "kube-api-access-fkx8j") pod "45ca14f1-53eb-4bae-9610-3258ec585039" (UID: "45ca14f1-53eb-4bae-9610-3258ec585039"). InnerVolumeSpecName "kube-api-access-fkx8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:13:55.532874 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.532838    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:13:55.532874 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.532869    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkx8j\" (UniqueName: \"kubernetes.io/projected/45ca14f1-53eb-4bae-9610-3258ec585039-kube-api-access-fkx8j\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:13:55.532874 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.532882    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ca14f1-53eb-4bae-9610-3258ec585039-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:13:55.533143 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.532894    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7966b\" (UniqueName: \"kubernetes.io/projected/1e9081f6-4035-4074-af23-11bd22ddce24-kube-api-access-7966b\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:13:55.533143 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.532904    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e9081f6-4035-4074-af23-11bd22ddce24-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:13:55.533143 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.532912    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ca14f1-53eb-4bae-9610-3258ec585039-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:13:55.727740 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.727706    2578 generic.go:356] "Generic (PLEG): container finished" podID="1e9081f6-4035-4074-af23-11bd22ddce24" containerID="788da2bffe508d7e135de8e17c88f91e5d545e6eb4ae22613c3225cf50b8562d" exitCode=0
Sep 18 20:13:55.727923 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.727754    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"
Sep 18 20:13:55.727923 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.727791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r" event={"ID":"1e9081f6-4035-4074-af23-11bd22ddce24","Type":"ContainerDied","Data":"788da2bffe508d7e135de8e17c88f91e5d545e6eb4ae22613c3225cf50b8562d"}
Sep 18 20:13:55.727923 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.727850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r" event={"ID":"1e9081f6-4035-4074-af23-11bd22ddce24","Type":"ContainerDied","Data":"7f8e1ebe52221240600451ca2ed6c2f946defb924b372ffbab1e9a1e1d04c6d4"}
Sep 18 20:13:55.727923 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.727884    2578 scope.go:117] "RemoveContainer" containerID="788da2bffe508d7e135de8e17c88f91e5d545e6eb4ae22613c3225cf50b8562d"
Sep 18 20:13:55.729578 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.729557    2578 generic.go:356] "Generic (PLEG): container finished" podID="45ca14f1-53eb-4bae-9610-3258ec585039" containerID="6c0eb77dce3a7f11f76557f18bbeb28713810b4acb05aaed836791d36dbb9158" exitCode=0
Sep 18 20:13:55.729698 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.729583    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r" event={"ID":"45ca14f1-53eb-4bae-9610-3258ec585039","Type":"ContainerDied","Data":"6c0eb77dce3a7f11f76557f18bbeb28713810b4acb05aaed836791d36dbb9158"}
Sep 18 20:13:55.729698 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.729613    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"
Sep 18 20:13:55.729698 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.729628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r" event={"ID":"45ca14f1-53eb-4bae-9610-3258ec585039","Type":"ContainerDied","Data":"d575bc4cf47e5be160bc1585abf308f1ea3e48d419845a7a788f6e2b20fcfe20"}
Sep 18 20:13:55.738435 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.738422    2578 scope.go:117] "RemoveContainer" containerID="788da2bffe508d7e135de8e17c88f91e5d545e6eb4ae22613c3225cf50b8562d"
Sep 18 20:13:55.738717 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:13:55.738699    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788da2bffe508d7e135de8e17c88f91e5d545e6eb4ae22613c3225cf50b8562d\": container with ID starting with 788da2bffe508d7e135de8e17c88f91e5d545e6eb4ae22613c3225cf50b8562d not found: ID does not exist" containerID="788da2bffe508d7e135de8e17c88f91e5d545e6eb4ae22613c3225cf50b8562d"
Sep 18 20:13:55.738772 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.738725    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788da2bffe508d7e135de8e17c88f91e5d545e6eb4ae22613c3225cf50b8562d"} err="failed to get container status \"788da2bffe508d7e135de8e17c88f91e5d545e6eb4ae22613c3225cf50b8562d\": rpc error: code = NotFound desc = could not find container \"788da2bffe508d7e135de8e17c88f91e5d545e6eb4ae22613c3225cf50b8562d\": container with ID starting with 788da2bffe508d7e135de8e17c88f91e5d545e6eb4ae22613c3225cf50b8562d not found: ID does not exist"
Sep 18 20:13:55.738772 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.738742    2578 scope.go:117] "RemoveContainer" containerID="6c0eb77dce3a7f11f76557f18bbeb28713810b4acb05aaed836791d36dbb9158"
Sep 18 20:13:55.746755 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.746739    2578 scope.go:117] "RemoveContainer" containerID="6c0eb77dce3a7f11f76557f18bbeb28713810b4acb05aaed836791d36dbb9158"
Sep 18 20:13:55.747026 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:13:55.747007    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0eb77dce3a7f11f76557f18bbeb28713810b4acb05aaed836791d36dbb9158\": container with ID starting with 6c0eb77dce3a7f11f76557f18bbeb28713810b4acb05aaed836791d36dbb9158 not found: ID does not exist" containerID="6c0eb77dce3a7f11f76557f18bbeb28713810b4acb05aaed836791d36dbb9158"
Sep 18 20:13:55.747075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.747032    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0eb77dce3a7f11f76557f18bbeb28713810b4acb05aaed836791d36dbb9158"} err="failed to get container status \"6c0eb77dce3a7f11f76557f18bbeb28713810b4acb05aaed836791d36dbb9158\": rpc error: code = NotFound desc = could not find container \"6c0eb77dce3a7f11f76557f18bbeb28713810b4acb05aaed836791d36dbb9158\": container with ID starting with 6c0eb77dce3a7f11f76557f18bbeb28713810b4acb05aaed836791d36dbb9158 not found: ID does not exist"
Sep 18 20:13:55.752375 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.752356    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"]
Sep 18 20:13:55.755863 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.755840    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f524d4ec-9d7c9c89c-pb75r"]
Sep 18 20:13:55.769889 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.769871    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"]
Sep 18 20:13:55.777485 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.777464    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f524d4ec-56c8bc7b6d-k7g8r"]
Sep 18 20:13:55.813431 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:55.813399    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:56.319344 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:56.319316    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e9081f6-4035-4074-af23-11bd22ddce24" path="/var/lib/kubelet/pods/1e9081f6-4035-4074-af23-11bd22ddce24/volumes"
Sep 18 20:13:56.319702 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:56.319674    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ca14f1-53eb-4bae-9610-3258ec585039" path="/var/lib/kubelet/pods/45ca14f1-53eb-4bae-9610-3258ec585039/volumes"
Sep 18 20:13:56.519148 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:56.519104    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:56.914090 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:56.913982    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:57.014361 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:57.014324    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:57.267372 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:57.267341    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:57.955384 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:57.955337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:58.029152 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:58.024280    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:58.814602 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:58.814567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:59.357620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:59.357583    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:59.455749 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:59.455714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:13:59.937456 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:13:59.937425    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:00.747724 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:00.747688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:01.704146 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.704115    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"]
Sep 18 20:14:01.704515 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.704500    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ca14f1-53eb-4bae-9610-3258ec585039" containerName="maas-api"
Sep 18 20:14:01.704573 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.704518    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ca14f1-53eb-4bae-9610-3258ec585039" containerName="maas-api"
Sep 18 20:14:01.704573 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.704536    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e9081f6-4035-4074-af23-11bd22ddce24" containerName="payload-processing"
Sep 18 20:14:01.704573 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.704541    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e9081f6-4035-4074-af23-11bd22ddce24" containerName="payload-processing"
Sep 18 20:14:01.704712 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.704606    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ca14f1-53eb-4bae-9610-3258ec585039" containerName="maas-api"
Sep 18 20:14:01.704712 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.704618    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e9081f6-4035-4074-af23-11bd22ddce24" containerName="payload-processing"
Sep 18 20:14:01.707684 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.707667    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:01.709925 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.709904    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-fa77d0d6-serving-cert\""
Sep 18 20:14:01.719271 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.719239    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"]
Sep 18 20:14:01.740920 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.740893    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:01.784392 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.784362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-openshift-service-ca\") pod \"maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:01.784735 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.784401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-metrics-certs\") pod \"maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:01.784735 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.784446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8995q\" (UniqueName: \"kubernetes.io/projected/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-kube-api-access-8995q\") pod \"maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:01.784735 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.784493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-maas-api-tls\") pod \"maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:01.814894 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.814863    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:01.885764 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.885721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-openshift-service-ca\") pod \"maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:01.885921 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.885787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-metrics-certs\") pod \"maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:01.885921 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.885823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8995q\" (UniqueName: \"kubernetes.io/projected/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-kube-api-access-8995q\") pod \"maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:01.885921 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.885864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-maas-api-tls\") pod \"maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:01.886080 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:14:01.885993    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-serving-cert: secret "maas-api-e2e-derive-fa77d0d6-serving-cert" not found
Sep 18 20:14:01.886080 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:14:01.886063    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-maas-api-tls podName:9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8 nodeName:}" failed. No retries permitted until 2026-09-18 20:14:02.386041256 +0000 UTC m=+1044.224505200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-maas-api-tls") pod "maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b" (UID: "9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8") : secret "maas-api-e2e-derive-fa77d0d6-serving-cert" not found
Sep 18 20:14:01.886507 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.886486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-openshift-service-ca\") pod \"maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:01.888285 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.888265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-metrics-certs\") pod \"maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:01.895846 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:01.895821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8995q\" (UniqueName: \"kubernetes.io/projected/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-kube-api-access-8995q\") pod \"maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:02.106703 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.106669    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:02.262284 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.262243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:02.360082 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.359983    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:02.388682 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.388658    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"]
Sep 18 20:14:02.389830 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.389812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-maas-api-tls\") pod \"maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:02.392258 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.392236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-maas-api-tls\") pod \"maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:02.392329 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.392293    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"
Sep 18 20:14:02.394871 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.394849    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-fa77d0d6\""
Sep 18 20:14:02.395005 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.394916    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-fa77d0d6-dockercfg-cr44t\""
Sep 18 20:14:02.404732 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.404709    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"]
Sep 18 20:14:02.442761 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.442734    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"]
Sep 18 20:14:02.446058 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.446037    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"
Sep 18 20:14:02.457510 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.457476    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"]
Sep 18 20:14:02.490849 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.490813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e26337a-42f4-4b10-b77b-ce3acb6111f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h\" (UID: \"5e26337a-42f4-4b10-b77b-ce3acb6111f6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"
Sep 18 20:14:02.491039 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.490859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp7zh\" (UniqueName: \"kubernetes.io/projected/5e26337a-42f4-4b10-b77b-ce3acb6111f6-kube-api-access-vp7zh\") pod \"payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h\" (UID: \"5e26337a-42f4-4b10-b77b-ce3acb6111f6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"
Sep 18 20:14:02.491039 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.490890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k5dx\" (UniqueName: \"kubernetes.io/projected/47062d62-2f67-4259-b987-ba4abbea129b-kube-api-access-6k5dx\") pod \"payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw\" (UID: \"47062d62-2f67-4259-b987-ba4abbea129b\") " pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"
Sep 18 20:14:02.491039 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.490988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47062d62-2f67-4259-b987-ba4abbea129b-plugins-config-volume\") pod \"payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw\" (UID: \"47062d62-2f67-4259-b987-ba4abbea129b\") " pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"
Sep 18 20:14:02.591741 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.591700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47062d62-2f67-4259-b987-ba4abbea129b-plugins-config-volume\") pod \"payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw\" (UID: \"47062d62-2f67-4259-b987-ba4abbea129b\") " pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"
Sep 18 20:14:02.591907 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.591839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e26337a-42f4-4b10-b77b-ce3acb6111f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h\" (UID: \"5e26337a-42f4-4b10-b77b-ce3acb6111f6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"
Sep 18 20:14:02.591907 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.591864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp7zh\" (UniqueName: \"kubernetes.io/projected/5e26337a-42f4-4b10-b77b-ce3acb6111f6-kube-api-access-vp7zh\") pod \"payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h\" (UID: \"5e26337a-42f4-4b10-b77b-ce3acb6111f6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"
Sep 18 20:14:02.591907 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.591892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k5dx\" (UniqueName: \"kubernetes.io/projected/47062d62-2f67-4259-b987-ba4abbea129b-kube-api-access-6k5dx\") pod \"payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw\" (UID: \"47062d62-2f67-4259-b987-ba4abbea129b\") " pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"
Sep 18 20:14:02.592451 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.592428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47062d62-2f67-4259-b987-ba4abbea129b-plugins-config-volume\") pod \"payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw\" (UID: \"47062d62-2f67-4259-b987-ba4abbea129b\") " pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"
Sep 18 20:14:02.592633 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.592615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e26337a-42f4-4b10-b77b-ce3acb6111f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h\" (UID: \"5e26337a-42f4-4b10-b77b-ce3acb6111f6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"
Sep 18 20:14:02.599848 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.599817    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp7zh\" (UniqueName: \"kubernetes.io/projected/5e26337a-42f4-4b10-b77b-ce3acb6111f6-kube-api-access-vp7zh\") pod \"payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h\" (UID: \"5e26337a-42f4-4b10-b77b-ce3acb6111f6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"
Sep 18 20:14:02.599974 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.599871    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k5dx\" (UniqueName: \"kubernetes.io/projected/47062d62-2f67-4259-b987-ba4abbea129b-kube-api-access-6k5dx\") pod \"payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw\" (UID: \"47062d62-2f67-4259-b987-ba4abbea129b\") " pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"
Sep 18 20:14:02.619449 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.619376    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:02.666953 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.666715    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:02.710862 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.710831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"
Sep 18 20:14:02.763528 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.763492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"
Sep 18 20:14:02.894023 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.894000    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"]
Sep 18 20:14:02.896523 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:14:02.896494    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd9eb2e_3f8a_426c_be00_68a1d7f7b1b8.slice/crio-783dc37f63853d81b3858bb61fcead810bfb0ccb6ab17557ad0131c930cb7964 WatchSource:0}: Error finding container 783dc37f63853d81b3858bb61fcead810bfb0ccb6ab17557ad0131c930cb7964: Status 404 returned error can't find the container with id 783dc37f63853d81b3858bb61fcead810bfb0ccb6ab17557ad0131c930cb7964
Sep 18 20:14:02.897832 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.897815    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:14:02.987544 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:02.987512    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"]
Sep 18 20:14:02.990850 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:14:02.990807    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47062d62_2f67_4259_b987_ba4abbea129b.slice/crio-fdb4cae5bc594f3a7a1d9aa250234b696f7140648108e2579eb30dc8a0eeaf16 WatchSource:0}: Error finding container fdb4cae5bc594f3a7a1d9aa250234b696f7140648108e2579eb30dc8a0eeaf16: Status 404 returned error can't find the container with id fdb4cae5bc594f3a7a1d9aa250234b696f7140648108e2579eb30dc8a0eeaf16
Sep 18 20:14:03.041897 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.041858    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"]
Sep 18 20:14:03.043493 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:14:03.043466    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e26337a_42f4_4b10_b77b_ce3acb6111f6.slice/crio-6aeb4e0c16132dbb8c9a53cde2acb51e3220ff772cf23b0a7ab95c11e5e0c151 WatchSource:0}: Error finding container 6aeb4e0c16132dbb8c9a53cde2acb51e3220ff772cf23b0a7ab95c11e5e0c151: Status 404 returned error can't find the container with id 6aeb4e0c16132dbb8c9a53cde2acb51e3220ff772cf23b0a7ab95c11e5e0c151
Sep 18 20:14:03.520664 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.520628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:03.775148 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.775055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h" event={"ID":"5e26337a-42f4-4b10-b77b-ce3acb6111f6","Type":"ContainerStarted","Data":"1a4703019c3ef69a3d7acbfa408185f1cb9e41ea3ab21516dc7f2be96a972858"}
Sep 18 20:14:03.775307 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.775229    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"
Sep 18 20:14:03.775307 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.775275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h" event={"ID":"5e26337a-42f4-4b10-b77b-ce3acb6111f6","Type":"ContainerStarted","Data":"6aeb4e0c16132dbb8c9a53cde2acb51e3220ff772cf23b0a7ab95c11e5e0c151"}
Sep 18 20:14:03.776796 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.776768    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw" event={"ID":"47062d62-2f67-4259-b987-ba4abbea129b","Type":"ContainerStarted","Data":"6cc0873aafd7efddfd893766fadbae2b85a6b7f3d3ab0eaca0340c634076f4ce"}
Sep 18 20:14:03.776947 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.776816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw" event={"ID":"47062d62-2f67-4259-b987-ba4abbea129b","Type":"ContainerStarted","Data":"fdb4cae5bc594f3a7a1d9aa250234b696f7140648108e2579eb30dc8a0eeaf16"}
Sep 18 20:14:03.776947 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.776872    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"
Sep 18 20:14:03.778450 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.778421    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b" event={"ID":"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8","Type":"ContainerStarted","Data":"1ecfac82c49cc137b02533b02b6638956f1112376fba90b1728dc134200baa5f"}
Sep 18 20:14:03.778559 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.778462    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:03.778559 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.778479    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b" event={"ID":"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8","Type":"ContainerStarted","Data":"783dc37f63853d81b3858bb61fcead810bfb0ccb6ab17557ad0131c930cb7964"}
Sep 18 20:14:03.805691 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.805649    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h" podStartSLOduration=1.805636637 podStartE2EDuration="1.805636637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:14:03.802963217 +0000 UTC m=+1045.641427166" watchObservedRunningTime="2026-09-18 20:14:03.805636637 +0000 UTC m=+1045.644100588"
Sep 18 20:14:03.831694 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.831640    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b" podStartSLOduration=2.232723704 podStartE2EDuration="2.831628206s" totalImagesPullingTime="598.904502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:14:03.82786724 +0000 UTC m=+1045.666331194" watchObservedRunningTime="2026-09-18 20:14:03.831628206 +0000 UTC m=+1045.670092162"
Sep 18 20:14:03.853510 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:03.853440    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw" podStartSLOduration=1.853419479 podStartE2EDuration="1.853419479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:14:03.851349167 +0000 UTC m=+1045.689813123" watchObservedRunningTime="2026-09-18 20:14:03.853419479 +0000 UTC m=+1045.691883432"
Sep 18 20:14:04.438573 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:04.438540    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:05.205736 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:05.205700    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:09.788567 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:09.788532    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:14.785743 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:14.785714    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"
Sep 18 20:14:14.786158 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:14.785783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"
Sep 18 20:14:20.846980 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:20.846925    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:26.930152 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:26.930110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:28.043133 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:28.043094    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:29.420073 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:29.420012    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:30.452630 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:30.452598    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:31.560066 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:31.560032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:32.684685 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:32.684659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:33.970194 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:33.970157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:35.144544 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:35.144517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:36.119441 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:36.119399    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:36.949401 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:36.949364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:37.753676 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:37.753636    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:37.760029 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:37.759986    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"]
Sep 18 20:14:37.760479 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:37.760284    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh" podUID="e11b92fd-5ff0-4474-9fc8-8bc63e38fa24" containerName="payload-processing" containerID="cri-o://e803de6a14ef37acc742a5b7b8ee0c94ebf749d2ce4c8e74568a7edd647998f6" gracePeriod=30
Sep 18 20:14:37.769849 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:37.769823    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"]
Sep 18 20:14:37.770144 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:37.770050    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn" podUID="956411fb-81d9-4c44-82c7-2825ef4f2aba" containerName="payload-pre-processing" containerID="cri-o://4d6f4b014aa0edd79d5f1ad4bfaf71c2d6120afa8255330ce0bede098e65babe" gracePeriod=30
Sep 18 20:14:37.858178 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:37.858134    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:37.956479 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:37.956438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:37.959650 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:37.959627    2578 generic.go:356] "Generic (PLEG): container finished" podID="e11b92fd-5ff0-4474-9fc8-8bc63e38fa24" containerID="e803de6a14ef37acc742a5b7b8ee0c94ebf749d2ce4c8e74568a7edd647998f6" exitCode=0
Sep 18 20:14:37.959869 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:37.959697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh" event={"ID":"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24","Type":"ContainerDied","Data":"e803de6a14ef37acc742a5b7b8ee0c94ebf749d2ce4c8e74568a7edd647998f6"}
Sep 18 20:14:37.961562 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:37.961542    2578 generic.go:356] "Generic (PLEG): container finished" podID="956411fb-81d9-4c44-82c7-2825ef4f2aba" containerID="4d6f4b014aa0edd79d5f1ad4bfaf71c2d6120afa8255330ce0bede098e65babe" exitCode=0
Sep 18 20:14:37.961663 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:37.961604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn" event={"ID":"956411fb-81d9-4c44-82c7-2825ef4f2aba","Type":"ContainerDied","Data":"4d6f4b014aa0edd79d5f1ad4bfaf71c2d6120afa8255330ce0bede098e65babe"}
Sep 18 20:14:38.043401 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.043365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:38.052254 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.052227    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"
Sep 18 20:14:38.055219 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.055200    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"
Sep 18 20:14:38.116160 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.116122    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956411fb-81d9-4c44-82c7-2825ef4f2aba-plugins-config-volume\") pod \"956411fb-81d9-4c44-82c7-2825ef4f2aba\" (UID: \"956411fb-81d9-4c44-82c7-2825ef4f2aba\") "
Sep 18 20:14:38.116347 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.116169    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e11b92fd-5ff0-4474-9fc8-8bc63e38fa24-plugins-config-volume\") pod \"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24\" (UID: \"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24\") "
Sep 18 20:14:38.116347 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.116220    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j879c\" (UniqueName: \"kubernetes.io/projected/956411fb-81d9-4c44-82c7-2825ef4f2aba-kube-api-access-j879c\") pod \"956411fb-81d9-4c44-82c7-2825ef4f2aba\" (UID: \"956411fb-81d9-4c44-82c7-2825ef4f2aba\") "
Sep 18 20:14:38.116347 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.116321    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk96q\" (UniqueName: \"kubernetes.io/projected/e11b92fd-5ff0-4474-9fc8-8bc63e38fa24-kube-api-access-tk96q\") pod \"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24\" (UID: \"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24\") "
Sep 18 20:14:38.116673 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.116586    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956411fb-81d9-4c44-82c7-2825ef4f2aba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "956411fb-81d9-4c44-82c7-2825ef4f2aba" (UID: "956411fb-81d9-4c44-82c7-2825ef4f2aba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:38.116673 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.116592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11b92fd-5ff0-4474-9fc8-8bc63e38fa24-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e11b92fd-5ff0-4474-9fc8-8bc63e38fa24" (UID: "e11b92fd-5ff0-4474-9fc8-8bc63e38fa24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:38.118695 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.118670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956411fb-81d9-4c44-82c7-2825ef4f2aba-kube-api-access-j879c" (OuterVolumeSpecName: "kube-api-access-j879c") pod "956411fb-81d9-4c44-82c7-2825ef4f2aba" (UID: "956411fb-81d9-4c44-82c7-2825ef4f2aba"). InnerVolumeSpecName "kube-api-access-j879c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:38.119028 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.119000    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11b92fd-5ff0-4474-9fc8-8bc63e38fa24-kube-api-access-tk96q" (OuterVolumeSpecName: "kube-api-access-tk96q") pod "e11b92fd-5ff0-4474-9fc8-8bc63e38fa24" (UID: "e11b92fd-5ff0-4474-9fc8-8bc63e38fa24"). InnerVolumeSpecName "kube-api-access-tk96q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:38.217794 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.217764    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956411fb-81d9-4c44-82c7-2825ef4f2aba-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:14:38.217794 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.217788    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e11b92fd-5ff0-4474-9fc8-8bc63e38fa24-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:14:38.217794 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.217797    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j879c\" (UniqueName: \"kubernetes.io/projected/956411fb-81d9-4c44-82c7-2825ef4f2aba-kube-api-access-j879c\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:14:38.218030 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.217807    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk96q\" (UniqueName: \"kubernetes.io/projected/e11b92fd-5ff0-4474-9fc8-8bc63e38fa24-kube-api-access-tk96q\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:14:38.353920 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.353891    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:38.450363 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.450328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:38.968163 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.968117    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"
Sep 18 20:14:38.968163 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.968162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn" event={"ID":"956411fb-81d9-4c44-82c7-2825ef4f2aba","Type":"ContainerDied","Data":"93563270c1b65819709dd5140d17c946cbf585d083cd9b6e2ea581d485e68780"}
Sep 18 20:14:38.968672 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.968230    2578 scope.go:117] "RemoveContainer" containerID="4d6f4b014aa0edd79d5f1ad4bfaf71c2d6120afa8255330ce0bede098e65babe"
Sep 18 20:14:38.969765 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.969615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh" event={"ID":"e11b92fd-5ff0-4474-9fc8-8bc63e38fa24","Type":"ContainerDied","Data":"43bff496d1e3a486d14c5f658ed5a00a44e37f5d483fa2a414a3885e97b415ff"}
Sep 18 20:14:38.969765 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.969620    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"
Sep 18 20:14:38.977489 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.977470    2578 scope.go:117] "RemoveContainer" containerID="e803de6a14ef37acc742a5b7b8ee0c94ebf749d2ce4c8e74568a7edd647998f6"
Sep 18 20:14:38.987791 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.987772    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"]
Sep 18 20:14:38.992224 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:38.992192    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d0d77c-649b889959-bkvfh"]
Sep 18 20:14:39.002372 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:39.002353    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"]
Sep 18 20:14:39.008137 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:39.008112    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d0d77c-68db6956bb-nxnvn"]
Sep 18 20:14:39.355165 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:39.355133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:40.154644 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:40.154605    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:40.325187 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:40.325159    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="956411fb-81d9-4c44-82c7-2825ef4f2aba" path="/var/lib/kubelet/pods/956411fb-81d9-4c44-82c7-2825ef4f2aba/volumes"
Sep 18 20:14:40.325674 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:40.325657    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e11b92fd-5ff0-4474-9fc8-8bc63e38fa24" path="/var/lib/kubelet/pods/e11b92fd-5ff0-4474-9fc8-8bc63e38fa24/volumes"
Sep 18 20:14:41.183599 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:41.183564    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:42.266209 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:42.266172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:43.147468 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:43.147431    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:43.521633 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:43.521596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:43.882422 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:43.882339    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"]
Sep 18 20:14:43.882617 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:43.882572    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b" podUID="9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8" containerName="maas-api" containerID="cri-o://1ecfac82c49cc137b02533b02b6638956f1112376fba90b1728dc134200baa5f" gracePeriod=30
Sep 18 20:14:43.927138 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:43.927109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:43.961868 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:43.961838    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"]
Sep 18 20:14:43.962075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:43.962048    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw" podUID="47062d62-2f67-4259-b987-ba4abbea129b" containerName="payload-processing" containerID="cri-o://6cc0873aafd7efddfd893766fadbae2b85a6b7f3d3ab0eaca0340c634076f4ce" gracePeriod=30
Sep 18 20:14:43.990637 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:43.990607    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"]
Sep 18 20:14:43.990857 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:43.990831    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h" podUID="5e26337a-42f4-4b10-b77b-ce3acb6111f6" containerName="payload-pre-processing" containerID="cri-o://1a4703019c3ef69a3d7acbfa408185f1cb9e41ea3ab21516dc7f2be96a972858" gracePeriod=30
Sep 18 20:14:44.000008 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:43.999982    2578 generic.go:356] "Generic (PLEG): container finished" podID="9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8" containerID="1ecfac82c49cc137b02533b02b6638956f1112376fba90b1728dc134200baa5f" exitCode=0
Sep 18 20:14:44.000145 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.000057    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b" event={"ID":"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8","Type":"ContainerDied","Data":"1ecfac82c49cc137b02533b02b6638956f1112376fba90b1728dc134200baa5f"}
Sep 18 20:14:44.028793 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.028765    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:44.140565 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.140490    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:44.219961 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.219910    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:44.266220 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.266195    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"
Sep 18 20:14:44.269544 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.269520    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"
Sep 18 20:14:44.269811 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.269795    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-maas-api-tls\") pod \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") "
Sep 18 20:14:44.269882 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.269837    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-openshift-service-ca\") pod \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") "
Sep 18 20:14:44.269959 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.269921    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-metrics-certs\") pod \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") "
Sep 18 20:14:44.270025 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.269986    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8995q\" (UniqueName: \"kubernetes.io/projected/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-kube-api-access-8995q\") pod \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\" (UID: \"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8\") "
Sep 18 20:14:44.270387 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.270356    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8" (UID: "9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:44.272022 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.271993    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8" (UID: "9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:14:44.272188 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.272167    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-kube-api-access-8995q" (OuterVolumeSpecName: "kube-api-access-8995q") pod "9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8" (UID: "9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8"). InnerVolumeSpecName "kube-api-access-8995q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:44.272434 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.272414    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8" (UID: "9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:14:44.291034 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.291003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:44.371197 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.371167    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47062d62-2f67-4259-b987-ba4abbea129b-plugins-config-volume\") pod \"47062d62-2f67-4259-b987-ba4abbea129b\" (UID: \"47062d62-2f67-4259-b987-ba4abbea129b\") "
Sep 18 20:14:44.371373 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.371248    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp7zh\" (UniqueName: \"kubernetes.io/projected/5e26337a-42f4-4b10-b77b-ce3acb6111f6-kube-api-access-vp7zh\") pod \"5e26337a-42f4-4b10-b77b-ce3acb6111f6\" (UID: \"5e26337a-42f4-4b10-b77b-ce3acb6111f6\") "
Sep 18 20:14:44.371373 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.371323    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e26337a-42f4-4b10-b77b-ce3acb6111f6-plugins-config-volume\") pod \"5e26337a-42f4-4b10-b77b-ce3acb6111f6\" (UID: \"5e26337a-42f4-4b10-b77b-ce3acb6111f6\") "
Sep 18 20:14:44.371373 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.371363    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k5dx\" (UniqueName: \"kubernetes.io/projected/47062d62-2f67-4259-b987-ba4abbea129b-kube-api-access-6k5dx\") pod \"47062d62-2f67-4259-b987-ba4abbea129b\" (UID: \"47062d62-2f67-4259-b987-ba4abbea129b\") "
Sep 18 20:14:44.371609 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.371589    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:14:44.371708 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.371615    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:14:44.371708 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.371629    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:14:44.371708 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.371645    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8995q\" (UniqueName: \"kubernetes.io/projected/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8-kube-api-access-8995q\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:14:44.371708 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.371669    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47062d62-2f67-4259-b987-ba4abbea129b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "47062d62-2f67-4259-b987-ba4abbea129b" (UID: "47062d62-2f67-4259-b987-ba4abbea129b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:44.372162 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.372088    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e26337a-42f4-4b10-b77b-ce3acb6111f6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e26337a-42f4-4b10-b77b-ce3acb6111f6" (UID: "5e26337a-42f4-4b10-b77b-ce3acb6111f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:44.374629 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.374605    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47062d62-2f67-4259-b987-ba4abbea129b-kube-api-access-6k5dx" (OuterVolumeSpecName: "kube-api-access-6k5dx") pod "47062d62-2f67-4259-b987-ba4abbea129b" (UID: "47062d62-2f67-4259-b987-ba4abbea129b"). InnerVolumeSpecName "kube-api-access-6k5dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:44.374727 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.374668    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e26337a-42f4-4b10-b77b-ce3acb6111f6-kube-api-access-vp7zh" (OuterVolumeSpecName: "kube-api-access-vp7zh") pod "5e26337a-42f4-4b10-b77b-ce3acb6111f6" (UID: "5e26337a-42f4-4b10-b77b-ce3acb6111f6"). InnerVolumeSpecName "kube-api-access-vp7zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:44.452081 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.451999    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:44.472161 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.472129    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e26337a-42f4-4b10-b77b-ce3acb6111f6-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:14:44.472161 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.472160    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k5dx\" (UniqueName: \"kubernetes.io/projected/47062d62-2f67-4259-b987-ba4abbea129b-kube-api-access-6k5dx\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:14:44.472298 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.472173    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47062d62-2f67-4259-b987-ba4abbea129b-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:14:44.472298 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.472185    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp7zh\" (UniqueName: \"kubernetes.io/projected/5e26337a-42f4-4b10-b77b-ce3acb6111f6-kube-api-access-vp7zh\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:14:44.739634 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:44.739601    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:45.007161 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.007086    2578 generic.go:356] "Generic (PLEG): container finished" podID="5e26337a-42f4-4b10-b77b-ce3acb6111f6" containerID="1a4703019c3ef69a3d7acbfa408185f1cb9e41ea3ab21516dc7f2be96a972858" exitCode=0
Sep 18 20:14:45.007161 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.007149    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"
Sep 18 20:14:45.007352 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.007163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h" event={"ID":"5e26337a-42f4-4b10-b77b-ce3acb6111f6","Type":"ContainerDied","Data":"1a4703019c3ef69a3d7acbfa408185f1cb9e41ea3ab21516dc7f2be96a972858"}
Sep 18 20:14:45.007352 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.007224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h" event={"ID":"5e26337a-42f4-4b10-b77b-ce3acb6111f6","Type":"ContainerDied","Data":"6aeb4e0c16132dbb8c9a53cde2acb51e3220ff772cf23b0a7ab95c11e5e0c151"}
Sep 18 20:14:45.007352 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.007251    2578 scope.go:117] "RemoveContainer" containerID="1a4703019c3ef69a3d7acbfa408185f1cb9e41ea3ab21516dc7f2be96a972858"
Sep 18 20:14:45.008587 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.008502    2578 generic.go:356] "Generic (PLEG): container finished" podID="47062d62-2f67-4259-b987-ba4abbea129b" containerID="6cc0873aafd7efddfd893766fadbae2b85a6b7f3d3ab0eaca0340c634076f4ce" exitCode=0
Sep 18 20:14:45.008587 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.008557    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"
Sep 18 20:14:45.008587 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.008578    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw" event={"ID":"47062d62-2f67-4259-b987-ba4abbea129b","Type":"ContainerDied","Data":"6cc0873aafd7efddfd893766fadbae2b85a6b7f3d3ab0eaca0340c634076f4ce"}
Sep 18 20:14:45.008754 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.008622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw" event={"ID":"47062d62-2f67-4259-b987-ba4abbea129b","Type":"ContainerDied","Data":"fdb4cae5bc594f3a7a1d9aa250234b696f7140648108e2579eb30dc8a0eeaf16"}
Sep 18 20:14:45.010216 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.010195    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"
Sep 18 20:14:45.010318 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.010224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b" event={"ID":"9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8","Type":"ContainerDied","Data":"783dc37f63853d81b3858bb61fcead810bfb0ccb6ab17557ad0131c930cb7964"}
Sep 18 20:14:45.017222 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.017204    2578 scope.go:117] "RemoveContainer" containerID="1a4703019c3ef69a3d7acbfa408185f1cb9e41ea3ab21516dc7f2be96a972858"
Sep 18 20:14:45.017475 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:14:45.017458    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4703019c3ef69a3d7acbfa408185f1cb9e41ea3ab21516dc7f2be96a972858\": container with ID starting with 1a4703019c3ef69a3d7acbfa408185f1cb9e41ea3ab21516dc7f2be96a972858 not found: ID does not exist" containerID="1a4703019c3ef69a3d7acbfa408185f1cb9e41ea3ab21516dc7f2be96a972858"
Sep 18 20:14:45.017543 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.017483    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4703019c3ef69a3d7acbfa408185f1cb9e41ea3ab21516dc7f2be96a972858"} err="failed to get container status \"1a4703019c3ef69a3d7acbfa408185f1cb9e41ea3ab21516dc7f2be96a972858\": rpc error: code = NotFound desc = could not find container \"1a4703019c3ef69a3d7acbfa408185f1cb9e41ea3ab21516dc7f2be96a972858\": container with ID starting with 1a4703019c3ef69a3d7acbfa408185f1cb9e41ea3ab21516dc7f2be96a972858 not found: ID does not exist"
Sep 18 20:14:45.017543 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.017499    2578 scope.go:117] "RemoveContainer" containerID="6cc0873aafd7efddfd893766fadbae2b85a6b7f3d3ab0eaca0340c634076f4ce"
Sep 18 20:14:45.025278 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.025258    2578 scope.go:117] "RemoveContainer" containerID="6cc0873aafd7efddfd893766fadbae2b85a6b7f3d3ab0eaca0340c634076f4ce"
Sep 18 20:14:45.025516 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:14:45.025497    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cc0873aafd7efddfd893766fadbae2b85a6b7f3d3ab0eaca0340c634076f4ce\": container with ID starting with 6cc0873aafd7efddfd893766fadbae2b85a6b7f3d3ab0eaca0340c634076f4ce not found: ID does not exist" containerID="6cc0873aafd7efddfd893766fadbae2b85a6b7f3d3ab0eaca0340c634076f4ce"
Sep 18 20:14:45.025561 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.025523    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cc0873aafd7efddfd893766fadbae2b85a6b7f3d3ab0eaca0340c634076f4ce"} err="failed to get container status \"6cc0873aafd7efddfd893766fadbae2b85a6b7f3d3ab0eaca0340c634076f4ce\": rpc error: code = NotFound desc = could not find container \"6cc0873aafd7efddfd893766fadbae2b85a6b7f3d3ab0eaca0340c634076f4ce\": container with ID starting with 6cc0873aafd7efddfd893766fadbae2b85a6b7f3d3ab0eaca0340c634076f4ce not found: ID does not exist"
Sep 18 20:14:45.025561 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.025540    2578 scope.go:117] "RemoveContainer" containerID="1ecfac82c49cc137b02533b02b6638956f1112376fba90b1728dc134200baa5f"
Sep 18 20:14:45.049547 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.049518    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"]
Sep 18 20:14:45.058559 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.058539    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fa77d0d6-56cc5dbc-mhp2b"]
Sep 18 20:14:45.109583 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.109556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:45.110573 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.110554    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"]
Sep 18 20:14:45.130580 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.130554    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-fa77d0d6-566cd99897-j79fw"]
Sep 18 20:14:45.158677 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.158648    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"]
Sep 18 20:14:45.164780 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.164756    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-fa77d0d6-549d6cd49d-xrc9h"]
Sep 18 20:14:45.237744 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.237705    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:45.836468 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:45.836436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:46.318242 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:46.318212    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47062d62-2f67-4259-b987-ba4abbea129b" path="/var/lib/kubelet/pods/47062d62-2f67-4259-b987-ba4abbea129b/volumes"
Sep 18 20:14:46.318580 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:46.318567    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e26337a-42f4-4b10-b77b-ce3acb6111f6" path="/var/lib/kubelet/pods/5e26337a-42f4-4b10-b77b-ce3acb6111f6/volumes"
Sep 18 20:14:46.318913 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:46.318902    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8" path="/var/lib/kubelet/pods/9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8/volumes"
Sep 18 20:14:46.649511 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:46.649426    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:46.800475 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:46.800432    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:47.723132 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:47.723102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:48.807757 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:48.807725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:48.906529 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:48.906490    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:49.744207 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:49.744155    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:50.554564 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:50.554530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:51.746869 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:51.746831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:52.785835 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:52.785806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:53.556956 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:53.556900    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:53.948228 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:53.948127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:54.069550 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:54.069511    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:54.188311 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:54.188275    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:54.279224 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:54.279178    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:54.655392 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:54.655309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:59.702600 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:59.702563    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:59.736586 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:59.736545    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"]
Sep 18 20:14:59.736848 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:59.736800    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84" podUID="e81f22d6-c572-4c5f-920f-661c9a82f430" containerName="payload-pre-processing" containerID="cri-o://4c05b4528baf1e105fc2e983f7634c71e0156a67b1a6f31318741fb410dcf28d" gracePeriod=30
Sep 18 20:14:59.806529 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:59.806497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:59.915713 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:59.915682    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:14:59.986744 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:14:59.986716    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"
Sep 18 20:15:00.068215 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.068174    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:00.085265 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.085232    2578 generic.go:356] "Generic (PLEG): container finished" podID="e81f22d6-c572-4c5f-920f-661c9a82f430" containerID="4c05b4528baf1e105fc2e983f7634c71e0156a67b1a6f31318741fb410dcf28d" exitCode=0
Sep 18 20:15:00.085419 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.085271    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"
Sep 18 20:15:00.085419 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.085309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84" event={"ID":"e81f22d6-c572-4c5f-920f-661c9a82f430","Type":"ContainerDied","Data":"4c05b4528baf1e105fc2e983f7634c71e0156a67b1a6f31318741fb410dcf28d"}
Sep 18 20:15:00.085419 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.085362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84" event={"ID":"e81f22d6-c572-4c5f-920f-661c9a82f430","Type":"ContainerDied","Data":"7746d8d5caf32509b7e118a426c5c9ff43875806a9e6e2a9c5634233493e35b9"}
Sep 18 20:15:00.085419 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.085393    2578 scope.go:117] "RemoveContainer" containerID="4c05b4528baf1e105fc2e983f7634c71e0156a67b1a6f31318741fb410dcf28d"
Sep 18 20:15:00.093623 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.093606    2578 scope.go:117] "RemoveContainer" containerID="4c05b4528baf1e105fc2e983f7634c71e0156a67b1a6f31318741fb410dcf28d"
Sep 18 20:15:00.093898 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:15:00.093879    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c05b4528baf1e105fc2e983f7634c71e0156a67b1a6f31318741fb410dcf28d\": container with ID starting with 4c05b4528baf1e105fc2e983f7634c71e0156a67b1a6f31318741fb410dcf28d not found: ID does not exist" containerID="4c05b4528baf1e105fc2e983f7634c71e0156a67b1a6f31318741fb410dcf28d"
Sep 18 20:15:00.093977 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.093909    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c05b4528baf1e105fc2e983f7634c71e0156a67b1a6f31318741fb410dcf28d"} err="failed to get container status \"4c05b4528baf1e105fc2e983f7634c71e0156a67b1a6f31318741fb410dcf28d\": rpc error: code = NotFound desc = could not find container \"4c05b4528baf1e105fc2e983f7634c71e0156a67b1a6f31318741fb410dcf28d\": container with ID starting with 4c05b4528baf1e105fc2e983f7634c71e0156a67b1a6f31318741fb410dcf28d not found: ID does not exist"
Sep 18 20:15:00.105748 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.105731    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpbrh\" (UniqueName: \"kubernetes.io/projected/e81f22d6-c572-4c5f-920f-661c9a82f430-kube-api-access-wpbrh\") pod \"e81f22d6-c572-4c5f-920f-661c9a82f430\" (UID: \"e81f22d6-c572-4c5f-920f-661c9a82f430\") "
Sep 18 20:15:00.105840 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.105760    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81f22d6-c572-4c5f-920f-661c9a82f430-plugins-config-volume\") pod \"e81f22d6-c572-4c5f-920f-661c9a82f430\" (UID: \"e81f22d6-c572-4c5f-920f-661c9a82f430\") "
Sep 18 20:15:00.106187 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.106165    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81f22d6-c572-4c5f-920f-661c9a82f430-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e81f22d6-c572-4c5f-920f-661c9a82f430" (UID: "e81f22d6-c572-4c5f-920f-661c9a82f430"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:15:00.107705 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.107684    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81f22d6-c572-4c5f-920f-661c9a82f430-kube-api-access-wpbrh" (OuterVolumeSpecName: "kube-api-access-wpbrh") pod "e81f22d6-c572-4c5f-920f-661c9a82f430" (UID: "e81f22d6-c572-4c5f-920f-661c9a82f430"). InnerVolumeSpecName "kube-api-access-wpbrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:15:00.193052 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193021    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5284dc-29829375-fmf6t"]
Sep 18 20:15:00.193357 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193345    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="956411fb-81d9-4c44-82c7-2825ef4f2aba" containerName="payload-pre-processing"
Sep 18 20:15:00.193401 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193359    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="956411fb-81d9-4c44-82c7-2825ef4f2aba" containerName="payload-pre-processing"
Sep 18 20:15:00.193401 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193370    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47062d62-2f67-4259-b987-ba4abbea129b" containerName="payload-processing"
Sep 18 20:15:00.193401 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193376    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47062d62-2f67-4259-b987-ba4abbea129b" containerName="payload-processing"
Sep 18 20:15:00.193401 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193390    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11b92fd-5ff0-4474-9fc8-8bc63e38fa24" containerName="payload-processing"
Sep 18 20:15:00.193401 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193395    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11b92fd-5ff0-4474-9fc8-8bc63e38fa24" containerName="payload-processing"
Sep 18 20:15:00.193558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193404    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e26337a-42f4-4b10-b77b-ce3acb6111f6" containerName="payload-pre-processing"
Sep 18 20:15:00.193558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193409    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e26337a-42f4-4b10-b77b-ce3acb6111f6" containerName="payload-pre-processing"
Sep 18 20:15:00.193558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193417    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e81f22d6-c572-4c5f-920f-661c9a82f430" containerName="payload-pre-processing"
Sep 18 20:15:00.193558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193422    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e81f22d6-c572-4c5f-920f-661c9a82f430" containerName="payload-pre-processing"
Sep 18 20:15:00.193558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193428    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8" containerName="maas-api"
Sep 18 20:15:00.193558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193433    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8" containerName="maas-api"
Sep 18 20:15:00.193558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193480    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="47062d62-2f67-4259-b987-ba4abbea129b" containerName="payload-processing"
Sep 18 20:15:00.193558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193488    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fd9eb2e-3f8a-426c-be00-68a1d7f7b1b8" containerName="maas-api"
Sep 18 20:15:00.193558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193494    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e81f22d6-c572-4c5f-920f-661c9a82f430" containerName="payload-pre-processing"
Sep 18 20:15:00.193558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193499    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="956411fb-81d9-4c44-82c7-2825ef4f2aba" containerName="payload-pre-processing"
Sep 18 20:15:00.193558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193505    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e26337a-42f4-4b10-b77b-ce3acb6111f6" containerName="payload-pre-processing"
Sep 18 20:15:00.193558 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.193513    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e11b92fd-5ff0-4474-9fc8-8bc63e38fa24" containerName="payload-processing"
Sep 18 20:15:00.197640 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.197622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5284dc-29829375-fmf6t"
Sep 18 20:15:00.201330 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.201310    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:15:00.209538 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.207051    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpbrh\" (UniqueName: \"kubernetes.io/projected/e81f22d6-c572-4c5f-920f-661c9a82f430-kube-api-access-wpbrh\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:15:00.209538 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.207081    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81f22d6-c572-4c5f-920f-661c9a82f430-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:15:00.210054 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.210027    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5284dc-29829375-fmf6t"
Sep 18 20:15:00.222997 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.222794    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5284dc-29829375-fmf6t"]
Sep 18 20:15:00.389737 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.389701    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:00.408228 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.408191    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"]
Sep 18 20:15:00.415609 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.415576    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4b633d-86ddb8f76b-bvk84"]
Sep 18 20:15:00.503784 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.503692    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5284dc-29829375-fmf6t"]
Sep 18 20:15:00.506708 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:15:00.506676    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88063197_3665_4a07_94e5_06aa35d058f7.slice/crio-399df27d63d1776aaee02a4fbe87d5b1bf2fc4d480123b88dc5d7cf86186e147 WatchSource:0}: Error finding container 399df27d63d1776aaee02a4fbe87d5b1bf2fc4d480123b88dc5d7cf86186e147: Status 404 returned error can't find the container with id 399df27d63d1776aaee02a4fbe87d5b1bf2fc4d480123b88dc5d7cf86186e147
Sep 18 20:15:00.659399 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:00.659364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:01.091949 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:01.091906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5284dc-29829375-fmf6t" event={"ID":"88063197-3665-4a07-94e5-06aa35d058f7","Type":"ContainerStarted","Data":"399df27d63d1776aaee02a4fbe87d5b1bf2fc4d480123b88dc5d7cf86186e147"}
Sep 18 20:15:01.866685 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:01.866650    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:02.320345 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:02.320306    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e81f22d6-c572-4c5f-920f-661c9a82f430" path="/var/lib/kubelet/pods/e81f22d6-c572-4c5f-920f-661c9a82f430/volumes"
Sep 18 20:15:02.931330 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:02.931299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:03.879052 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:03.879024    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:04.117240 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:04.117143    2578 generic.go:356] "Generic (PLEG): container finished" podID="88063197-3665-4a07-94e5-06aa35d058f7" containerID="ec033af44f3a02bdd6ce0709d9e594a31e29c3163d8ba714e50cd7ad23eb1e6e" exitCode=0
Sep 18 20:15:04.117240 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:04.117200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5284dc-29829375-fmf6t" event={"ID":"88063197-3665-4a07-94e5-06aa35d058f7","Type":"ContainerDied","Data":"ec033af44f3a02bdd6ce0709d9e594a31e29c3163d8ba714e50cd7ad23eb1e6e"}
Sep 18 20:15:05.240562 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:05.240534    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5284dc-29829375-fmf6t"
Sep 18 20:15:06.128294 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:06.128263    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5284dc-29829375-fmf6t"
Sep 18 20:15:06.128294 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:06.128285    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5284dc-29829375-fmf6t" event={"ID":"88063197-3665-4a07-94e5-06aa35d058f7","Type":"ContainerDied","Data":"399df27d63d1776aaee02a4fbe87d5b1bf2fc4d480123b88dc5d7cf86186e147"}
Sep 18 20:15:06.128495 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:06.128328    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="399df27d63d1776aaee02a4fbe87d5b1bf2fc4d480123b88dc5d7cf86186e147"
Sep 18 20:15:08.419486 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:08.419452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:08.521053 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:08.521022    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:08.716192 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:08.716161    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:16.217537 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.217508    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"]
Sep 18 20:15:16.217900 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.217829    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88063197-3665-4a07-94e5-06aa35d058f7" containerName="cleanup"
Sep 18 20:15:16.217900 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.217840    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88063197-3665-4a07-94e5-06aa35d058f7" containerName="cleanup"
Sep 18 20:15:16.217998 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.217919    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88063197-3665-4a07-94e5-06aa35d058f7" containerName="cleanup"
Sep 18 20:15:16.220787 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.220770    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.223594 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.223572    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5284dc\"/\"default-dockercfg-r2p8h\""
Sep 18 20:15:16.223722 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.223602    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5284dc\"/\"premium-w0-5284dc-kserve-self-signed-certs\""
Sep 18 20:15:16.223799 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.223782    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5284dc\"/\"openshift-service-ca.crt\""
Sep 18 20:15:16.224217 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.224200    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5284dc\"/\"kube-root-ca.crt\""
Sep 18 20:15:16.232338 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.232314    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"]
Sep 18 20:15:16.329665 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.329626    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:16.346681 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.346647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-home\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.346793 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.346690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk7f8\" (UniqueName: \"kubernetes.io/projected/4f109e24-48d7-4407-8b0a-18c022d517b4-kube-api-access-lk7f8\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.346793 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.346716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-tmp-dir\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.346793 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.346732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f109e24-48d7-4407-8b0a-18c022d517b4-tls-certs\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.346895 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.346805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-dshm\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.346895 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.346851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-model-cache\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.447543 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.447507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-tmp-dir\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.447543 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.447542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f109e24-48d7-4407-8b0a-18c022d517b4-tls-certs\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.447771 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.447586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-dshm\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.447771 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.447626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-model-cache\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.447771 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.447681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-home\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.447771 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.447715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7f8\" (UniqueName: \"kubernetes.io/projected/4f109e24-48d7-4407-8b0a-18c022d517b4-kube-api-access-lk7f8\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.448028 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.448007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-tmp-dir\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.448121 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.448099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-model-cache\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.448177 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.448137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-home\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.449883 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.449865    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-dshm\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.450302 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.450282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f109e24-48d7-4407-8b0a-18c022d517b4-tls-certs\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.458450 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.458427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7f8\" (UniqueName: \"kubernetes.io/projected/4f109e24-48d7-4407-8b0a-18c022d517b4-kube-api-access-lk7f8\") pod \"premium-w0-5284dc-kserve-565f857d8b-k9csg\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") " pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.529629 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.529544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:16.805052 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:16.805026    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"]
Sep 18 20:15:16.807074 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:15:16.807047    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f109e24_48d7_4407_8b0a_18c022d517b4.slice/crio-58a703f4802952881f6f44a5a51c458b885cfe7aed62ce370ad0db25b3153a46 WatchSource:0}: Error finding container 58a703f4802952881f6f44a5a51c458b885cfe7aed62ce370ad0db25b3153a46: Status 404 returned error can't find the container with id 58a703f4802952881f6f44a5a51c458b885cfe7aed62ce370ad0db25b3153a46
Sep 18 20:15:17.182600 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:17.182512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg" event={"ID":"4f109e24-48d7-4407-8b0a-18c022d517b4","Type":"ContainerStarted","Data":"7bd3743e6d4138e33509bd186b651c7d6967e21cc50524f8f295a12dfac3d558"}
Sep 18 20:15:17.182600 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:17.182562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg" event={"ID":"4f109e24-48d7-4407-8b0a-18c022d517b4","Type":"ContainerStarted","Data":"58a703f4802952881f6f44a5a51c458b885cfe7aed62ce370ad0db25b3153a46"}
Sep 18 20:15:17.203121 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:17.203076    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg" podStartSLOduration=1.203062162 podStartE2EDuration="1.203062162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:17.202049213 +0000 UTC m=+1119.040513188" watchObservedRunningTime="2026-09-18 20:15:17.203062162 +0000 UTC m=+1119.041526114"
Sep 18 20:15:20.559819 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:20.559787    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:20.660382 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:20.660346    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:20.815386 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:20.815299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:21.264185 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:21.264150    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:21.574625 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:21.574538    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:21.774128 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:21.774096    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:23.949966 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:23.949913    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:24.825617 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:24.825579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:24.956291 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:24.956255    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:25.636252 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:25.636217    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:26.529945 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:26.529889    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:26.530403 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:26.529965    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:26.542624 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:26.542600    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:26.975037 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:26.974994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:27.240861 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:27.240787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:15:27.935025 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:27.934989    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:28.404520 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:28.404492    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:28.512316 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:28.512277    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:28.632408 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:28.632375    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:28.872596 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:28.872561    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"]
Sep 18 20:15:28.875858 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:28.875840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:28.878072 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:28.878053    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-46f024-serving-cert\""
Sep 18 20:15:28.891078 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:28.891054    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"]
Sep 18 20:15:28.911191 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:28.911159    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:29.010624 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.010589    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:29.057496 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.057465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-maas-api-tls\") pod \"maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.057688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.057511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5789ca3-7407-4734-8bc3-9a3d48d6db75-openshift-service-ca\") pod \"maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.057688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.057601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5fxj\" (UniqueName: \"kubernetes.io/projected/e5789ca3-7407-4734-8bc3-9a3d48d6db75-kube-api-access-j5fxj\") pod \"maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.057688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.057634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-metrics-certs\") pod \"maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.159254 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.159165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-maas-api-tls\") pod \"maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.159254 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.159205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5789ca3-7407-4734-8bc3-9a3d48d6db75-openshift-service-ca\") pod \"maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.159254 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.159252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5fxj\" (UniqueName: \"kubernetes.io/projected/e5789ca3-7407-4734-8bc3-9a3d48d6db75-kube-api-access-j5fxj\") pod \"maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.159520 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.159275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-metrics-certs\") pod \"maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.159520 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:15:29.159326    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-serving-cert: secret "maas-api-e2e-explicit-46f024-serving-cert" not found
Sep 18 20:15:29.159520 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:15:29.159396    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-maas-api-tls podName:e5789ca3-7407-4734-8bc3-9a3d48d6db75 nodeName:}" failed. No retries permitted until 2026-09-18 20:15:29.659379784 +0000 UTC m=+1131.497843721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-maas-api-tls") pod "maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv" (UID: "e5789ca3-7407-4734-8bc3-9a3d48d6db75") : secret "maas-api-e2e-explicit-46f024-serving-cert" not found
Sep 18 20:15:29.160064 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.160035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5789ca3-7407-4734-8bc3-9a3d48d6db75-openshift-service-ca\") pod \"maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.161703 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.161686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-metrics-certs\") pod \"maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.170620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.170598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5fxj\" (UniqueName: \"kubernetes.io/projected/e5789ca3-7407-4734-8bc3-9a3d48d6db75-kube-api-access-j5fxj\") pod \"maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.317424 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.317394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:29.664769 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.664733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-maas-api-tls\") pod \"maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.667239 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.667211    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-maas-api-tls\") pod \"maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.720149 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.720108    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:29.786078 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.786041    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:29.915553 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:29.915375    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:30.053672 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.053637    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"]
Sep 18 20:15:30.057299 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.057272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"
Sep 18 20:15:30.060690 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.060649    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-46f024\""
Sep 18 20:15:30.060872 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.060858    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-46f024-dockercfg-v5xk2\""
Sep 18 20:15:30.073348 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.073322    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"]
Sep 18 20:15:30.077877 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.077836    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"]
Sep 18 20:15:30.080221 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:15:30.080199    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5789ca3_7407_4734_8bc3_9a3d48d6db75.slice/crio-1719efbc0dd51ef16474be84691afce9c708659d273de191088d0ac445e83bdf WatchSource:0}: Error finding container 1719efbc0dd51ef16474be84691afce9c708659d273de191088d0ac445e83bdf: Status 404 returned error can't find the container with id 1719efbc0dd51ef16474be84691afce9c708659d273de191088d0ac445e83bdf
Sep 18 20:15:30.169032 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.168957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpn67\" (UniqueName: \"kubernetes.io/projected/f16aede4-e2b4-4466-82c7-f32d2e7b5faa-kube-api-access-lpn67\") pod \"payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v\" (UID: \"f16aede4-e2b4-4466-82c7-f32d2e7b5faa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"
Sep 18 20:15:30.169173 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.169098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16aede4-e2b4-4466-82c7-f32d2e7b5faa-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v\" (UID: \"f16aede4-e2b4-4466-82c7-f32d2e7b5faa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"
Sep 18 20:15:30.244619 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.244584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv" event={"ID":"e5789ca3-7407-4734-8bc3-9a3d48d6db75","Type":"ContainerStarted","Data":"1719efbc0dd51ef16474be84691afce9c708659d273de191088d0ac445e83bdf"}
Sep 18 20:15:30.270493 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.270454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16aede4-e2b4-4466-82c7-f32d2e7b5faa-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v\" (UID: \"f16aede4-e2b4-4466-82c7-f32d2e7b5faa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"
Sep 18 20:15:30.270618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.270508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpn67\" (UniqueName: \"kubernetes.io/projected/f16aede4-e2b4-4466-82c7-f32d2e7b5faa-kube-api-access-lpn67\") pod \"payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v\" (UID: \"f16aede4-e2b4-4466-82c7-f32d2e7b5faa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"
Sep 18 20:15:30.271194 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.271175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16aede4-e2b4-4466-82c7-f32d2e7b5faa-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v\" (UID: \"f16aede4-e2b4-4466-82c7-f32d2e7b5faa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"
Sep 18 20:15:30.278849 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.278830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpn67\" (UniqueName: \"kubernetes.io/projected/f16aede4-e2b4-4466-82c7-f32d2e7b5faa-kube-api-access-lpn67\") pod \"payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v\" (UID: \"f16aede4-e2b4-4466-82c7-f32d2e7b5faa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"
Sep 18 20:15:30.367538 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.367503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"
Sep 18 20:15:30.641901 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:30.641864    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"]
Sep 18 20:15:30.647334 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:15:30.647308    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16aede4_e2b4_4466_82c7_f32d2e7b5faa.slice/crio-617a3b84599b745795e70e62d42fefc33a5a404fd9e1da2447af48e678ff3589 WatchSource:0}: Error finding container 617a3b84599b745795e70e62d42fefc33a5a404fd9e1da2447af48e678ff3589: Status 404 returned error can't find the container with id 617a3b84599b745795e70e62d42fefc33a5a404fd9e1da2447af48e678ff3589
Sep 18 20:15:31.251484 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:31.251448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v" event={"ID":"f16aede4-e2b4-4466-82c7-f32d2e7b5faa","Type":"ContainerStarted","Data":"3778cab3d729a3e7cc3b5f3c3652421381affd3945f4833f3bd4a22db571fb7c"}
Sep 18 20:15:31.251917 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:31.251517    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"
Sep 18 20:15:31.251917 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:31.251544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v" event={"ID":"f16aede4-e2b4-4466-82c7-f32d2e7b5faa","Type":"ContainerStarted","Data":"617a3b84599b745795e70e62d42fefc33a5a404fd9e1da2447af48e678ff3589"}
Sep 18 20:15:31.253015 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:31.252977    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv" event={"ID":"e5789ca3-7407-4734-8bc3-9a3d48d6db75","Type":"ContainerStarted","Data":"5ff6f58d532d93c9a8cc240facec687f63221e3ed5e1c66550197526f3f39a46"}
Sep 18 20:15:31.253130 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:31.253091    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:31.270737 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:31.270692    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v" podStartSLOduration=1.270677948 podStartE2EDuration="1.270677948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:31.267123338 +0000 UTC m=+1133.105587291" watchObservedRunningTime="2026-09-18 20:15:31.270677948 +0000 UTC m=+1133.109141901"
Sep 18 20:15:31.288318 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:31.288271    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv" podStartSLOduration=2.923913164 podStartE2EDuration="3.288252021s" totalImagesPullingTime="364.338857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:31.284883789 +0000 UTC m=+1133.123347741" watchObservedRunningTime="2026-09-18 20:15:31.288252021 +0000 UTC m=+1133.126715977"
Sep 18 20:15:32.051988 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:32.051957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:32.189656 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:32.189618    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:37.264329 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:37.264296    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:15:38.727634 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:38.727600    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:42.261233 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:42.261205    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"
Sep 18 20:15:44.496067 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:44.496023    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:50.834666 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:50.834633    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:50.949850 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:50.949819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:51.116698 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:51.116617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:52.874919 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:52.874830    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:53.693197 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:53.693161    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:54.619058 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:54.619019    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:54.718691 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:54.718656    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:55.737386 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:55.737356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:56.599572 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:56.599532    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:57.197133 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:57.197101    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:15:57.301497 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:15:57.301461    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:01.589744 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:01.589703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:02.364113 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:02.364085    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:04.374771 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:04.374738    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:08.761686 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:08.761645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:08.866527 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:08.866491    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:08.976432 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:08.976383    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:09.217749 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:09.217721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:09.821104 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:09.821063    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:09.986959 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:09.986896    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:10.206521 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.206437    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:10.336477 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.336446    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"]
Sep 18 20:16:10.340027 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.340010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.343544 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.343523    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-46f024\"/\"default-dockercfg-lbjwk\""
Sep 18 20:16:10.343970 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.343954    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-46f024\"/\"openshift-service-ca.crt\""
Sep 18 20:16:10.344298 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.344283    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-46f024\"/\"explicit-model-46f024-kserve-self-signed-certs\""
Sep 18 20:16:10.344391 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.344372    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-46f024\"/\"kube-root-ca.crt\""
Sep 18 20:16:10.361271 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.361245    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"]
Sep 18 20:16:10.408845 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.408809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:10.416016 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.415982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-model-cache\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.416145 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.416040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-tmp-dir\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.416208 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.416182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-dshm\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.416285 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.416262    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53e2d41-22ef-41a0-b613-e2c836fbc407-tls-certs\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.416350 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.416333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gttk8\" (UniqueName: \"kubernetes.io/projected/f53e2d41-22ef-41a0-b613-e2c836fbc407-kube-api-access-gttk8\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.416413 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.416358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-home\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.517405 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.517369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-model-cache\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.517581 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.517412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-tmp-dir\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.517581 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.517447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-dshm\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.517581 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.517474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53e2d41-22ef-41a0-b613-e2c836fbc407-tls-certs\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.517581 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.517508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gttk8\" (UniqueName: \"kubernetes.io/projected/f53e2d41-22ef-41a0-b613-e2c836fbc407-kube-api-access-gttk8\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.517805 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.517602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-home\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.517865 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.517799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-model-cache\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.517865 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.517853    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-tmp-dir\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.517991 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.517927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-home\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.519810 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.519789    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-dshm\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.520095 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.520071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53e2d41-22ef-41a0-b613-e2c836fbc407-tls-certs\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.526203 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.526169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gttk8\" (UniqueName: \"kubernetes.io/projected/f53e2d41-22ef-41a0-b613-e2c836fbc407-kube-api-access-gttk8\") pod \"explicit-model-46f024-kserve-7d879f6b49-t6hqd\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") " pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.649145 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.649112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:10.934618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:10.934586    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"]
Sep 18 20:16:10.936227 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:16:10.936201    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf53e2d41_22ef_41a0_b613_e2c836fbc407.slice/crio-16da380b205af1f84ccb75ae80d0db56255f9cdc5761a39ca0087b584b68d332 WatchSource:0}: Error finding container 16da380b205af1f84ccb75ae80d0db56255f9cdc5761a39ca0087b584b68d332: Status 404 returned error can't find the container with id 16da380b205af1f84ccb75ae80d0db56255f9cdc5761a39ca0087b584b68d332
Sep 18 20:16:11.455279 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:11.455240    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd" event={"ID":"f53e2d41-22ef-41a0-b613-e2c836fbc407","Type":"ContainerStarted","Data":"9a9eaa23c75e86e73321f566f266f1fbc6fede8aeca3cfbbbe67dcbe9d377da2"}
Sep 18 20:16:11.455279 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:11.455282    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd" event={"ID":"f53e2d41-22ef-41a0-b613-e2c836fbc407","Type":"ContainerStarted","Data":"16da380b205af1f84ccb75ae80d0db56255f9cdc5761a39ca0087b584b68d332"}
Sep 18 20:16:11.476101 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:11.476051    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd" podStartSLOduration=1.476036371 podStartE2EDuration="1.476036371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:11.474778936 +0000 UTC m=+1173.313242892" watchObservedRunningTime="2026-09-18 20:16:11.476036371 +0000 UTC m=+1173.314500322"
Sep 18 20:16:11.549606 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:11.549573    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:13.609387 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:13.609353    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"]
Sep 18 20:16:13.628167 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:13.628129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:14.752169 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:14.752135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:16.076250 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:16.076207    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:17.180957 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:17.180909    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:18.685864 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:18.685831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:20.112760 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:20.112683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:20.649440 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:20.649408    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:20.803620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:20.803584    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:20.933016 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:20.932944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:21.541528 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:21.541497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:24.864465 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:24.864433    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"]
Sep 18 20:16:24.864842 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:24.864660    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv" podUID="e5789ca3-7407-4734-8bc3-9a3d48d6db75" containerName="maas-api" containerID="cri-o://5ff6f58d532d93c9a8cc240facec687f63221e3ed5e1c66550197526f3f39a46" gracePeriod=30
Sep 18 20:16:24.964118 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:24.964070    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:25.049160 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.049131    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"]
Sep 18 20:16:25.049364 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.049337    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v" podUID="f16aede4-e2b4-4466-82c7-f32d2e7b5faa" containerName="payload-pre-processing" containerID="cri-o://3778cab3d729a3e7cc3b5f3c3652421381affd3945f4833f3bd4a22db571fb7c" gracePeriod=30
Sep 18 20:16:25.139546 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.139520    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:25.175123 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.175095    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:16:25.246884 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.246850    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-metrics-certs\") pod \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") "
Sep 18 20:16:25.247046 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.246902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5fxj\" (UniqueName: \"kubernetes.io/projected/e5789ca3-7407-4734-8bc3-9a3d48d6db75-kube-api-access-j5fxj\") pod \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") "
Sep 18 20:16:25.247046 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.246963    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-maas-api-tls\") pod \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") "
Sep 18 20:16:25.247046 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.246993    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5789ca3-7407-4734-8bc3-9a3d48d6db75-openshift-service-ca\") pod \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\" (UID: \"e5789ca3-7407-4734-8bc3-9a3d48d6db75\") "
Sep 18 20:16:25.247404 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.247373    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5789ca3-7407-4734-8bc3-9a3d48d6db75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5789ca3-7407-4734-8bc3-9a3d48d6db75" (UID: "e5789ca3-7407-4734-8bc3-9a3d48d6db75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:16:25.249139 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.249108    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e5789ca3-7407-4734-8bc3-9a3d48d6db75" (UID: "e5789ca3-7407-4734-8bc3-9a3d48d6db75"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:16:25.249480 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.249452    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5789ca3-7407-4734-8bc3-9a3d48d6db75" (UID: "e5789ca3-7407-4734-8bc3-9a3d48d6db75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:16:25.249480 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.249462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5789ca3-7407-4734-8bc3-9a3d48d6db75-kube-api-access-j5fxj" (OuterVolumeSpecName: "kube-api-access-j5fxj") pod "e5789ca3-7407-4734-8bc3-9a3d48d6db75" (UID: "e5789ca3-7407-4734-8bc3-9a3d48d6db75"). InnerVolumeSpecName "kube-api-access-j5fxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:25.298493 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.298468    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"
Sep 18 20:16:25.348288 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.348260    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16aede4-e2b4-4466-82c7-f32d2e7b5faa-plugins-config-volume\") pod \"f16aede4-e2b4-4466-82c7-f32d2e7b5faa\" (UID: \"f16aede4-e2b4-4466-82c7-f32d2e7b5faa\") "
Sep 18 20:16:25.348440 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.348308    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpn67\" (UniqueName: \"kubernetes.io/projected/f16aede4-e2b4-4466-82c7-f32d2e7b5faa-kube-api-access-lpn67\") pod \"f16aede4-e2b4-4466-82c7-f32d2e7b5faa\" (UID: \"f16aede4-e2b4-4466-82c7-f32d2e7b5faa\") "
Sep 18 20:16:25.348625 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.348607    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:16:25.348674 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.348632    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5fxj\" (UniqueName: \"kubernetes.io/projected/e5789ca3-7407-4734-8bc3-9a3d48d6db75-kube-api-access-j5fxj\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:16:25.348674 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.348649    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5789ca3-7407-4734-8bc3-9a3d48d6db75-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:16:25.348674 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.348664    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5789ca3-7407-4734-8bc3-9a3d48d6db75-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:16:25.348768 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.348670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16aede4-e2b4-4466-82c7-f32d2e7b5faa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f16aede4-e2b4-4466-82c7-f32d2e7b5faa" (UID: "f16aede4-e2b4-4466-82c7-f32d2e7b5faa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:16:25.350461 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.350441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16aede4-e2b4-4466-82c7-f32d2e7b5faa-kube-api-access-lpn67" (OuterVolumeSpecName: "kube-api-access-lpn67") pod "f16aede4-e2b4-4466-82c7-f32d2e7b5faa" (UID: "f16aede4-e2b4-4466-82c7-f32d2e7b5faa"). InnerVolumeSpecName "kube-api-access-lpn67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:25.356216 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.356182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:25.449201 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.449121    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16aede4-e2b4-4466-82c7-f32d2e7b5faa-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:16:25.449201 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.449150    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpn67\" (UniqueName: \"kubernetes.io/projected/f16aede4-e2b4-4466-82c7-f32d2e7b5faa-kube-api-access-lpn67\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:16:25.524262 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.524229    2578 generic.go:356] "Generic (PLEG): container finished" podID="f16aede4-e2b4-4466-82c7-f32d2e7b5faa" containerID="3778cab3d729a3e7cc3b5f3c3652421381affd3945f4833f3bd4a22db571fb7c" exitCode=0
Sep 18 20:16:25.524447 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.524273    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"
Sep 18 20:16:25.524447 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.524313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v" event={"ID":"f16aede4-e2b4-4466-82c7-f32d2e7b5faa","Type":"ContainerDied","Data":"3778cab3d729a3e7cc3b5f3c3652421381affd3945f4833f3bd4a22db571fb7c"}
Sep 18 20:16:25.524447 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.524376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v" event={"ID":"f16aede4-e2b4-4466-82c7-f32d2e7b5faa","Type":"ContainerDied","Data":"617a3b84599b745795e70e62d42fefc33a5a404fd9e1da2447af48e678ff3589"}
Sep 18 20:16:25.524447 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.524409    2578 scope.go:117] "RemoveContainer" containerID="3778cab3d729a3e7cc3b5f3c3652421381affd3945f4833f3bd4a22db571fb7c"
Sep 18 20:16:25.525776 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.525753    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5789ca3-7407-4734-8bc3-9a3d48d6db75" containerID="5ff6f58d532d93c9a8cc240facec687f63221e3ed5e1c66550197526f3f39a46" exitCode=0
Sep 18 20:16:25.525897 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.525786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv" event={"ID":"e5789ca3-7407-4734-8bc3-9a3d48d6db75","Type":"ContainerDied","Data":"5ff6f58d532d93c9a8cc240facec687f63221e3ed5e1c66550197526f3f39a46"}
Sep 18 20:16:25.525897 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.525817    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"
Sep 18 20:16:25.525897 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.525846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv" event={"ID":"e5789ca3-7407-4734-8bc3-9a3d48d6db75","Type":"ContainerDied","Data":"1719efbc0dd51ef16474be84691afce9c708659d273de191088d0ac445e83bdf"}
Sep 18 20:16:25.534414 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.534299    2578 scope.go:117] "RemoveContainer" containerID="3778cab3d729a3e7cc3b5f3c3652421381affd3945f4833f3bd4a22db571fb7c"
Sep 18 20:16:25.534648 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:16:25.534622    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3778cab3d729a3e7cc3b5f3c3652421381affd3945f4833f3bd4a22db571fb7c\": container with ID starting with 3778cab3d729a3e7cc3b5f3c3652421381affd3945f4833f3bd4a22db571fb7c not found: ID does not exist" containerID="3778cab3d729a3e7cc3b5f3c3652421381affd3945f4833f3bd4a22db571fb7c"
Sep 18 20:16:25.534787 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.534655    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3778cab3d729a3e7cc3b5f3c3652421381affd3945f4833f3bd4a22db571fb7c"} err="failed to get container status \"3778cab3d729a3e7cc3b5f3c3652421381affd3945f4833f3bd4a22db571fb7c\": rpc error: code = NotFound desc = could not find container \"3778cab3d729a3e7cc3b5f3c3652421381affd3945f4833f3bd4a22db571fb7c\": container with ID starting with 3778cab3d729a3e7cc3b5f3c3652421381affd3945f4833f3bd4a22db571fb7c not found: ID does not exist"
Sep 18 20:16:25.534787 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.534676    2578 scope.go:117] "RemoveContainer" containerID="5ff6f58d532d93c9a8cc240facec687f63221e3ed5e1c66550197526f3f39a46"
Sep 18 20:16:25.543826 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.543801    2578 scope.go:117] "RemoveContainer" containerID="5ff6f58d532d93c9a8cc240facec687f63221e3ed5e1c66550197526f3f39a46"
Sep 18 20:16:25.544282 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:16:25.544216    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff6f58d532d93c9a8cc240facec687f63221e3ed5e1c66550197526f3f39a46\": container with ID starting with 5ff6f58d532d93c9a8cc240facec687f63221e3ed5e1c66550197526f3f39a46 not found: ID does not exist" containerID="5ff6f58d532d93c9a8cc240facec687f63221e3ed5e1c66550197526f3f39a46"
Sep 18 20:16:25.544282 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.544249    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff6f58d532d93c9a8cc240facec687f63221e3ed5e1c66550197526f3f39a46"} err="failed to get container status \"5ff6f58d532d93c9a8cc240facec687f63221e3ed5e1c66550197526f3f39a46\": rpc error: code = NotFound desc = could not find container \"5ff6f58d532d93c9a8cc240facec687f63221e3ed5e1c66550197526f3f39a46\": container with ID starting with 5ff6f58d532d93c9a8cc240facec687f63221e3ed5e1c66550197526f3f39a46 not found: ID does not exist"
Sep 18 20:16:25.568726 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.568697    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"]
Sep 18 20:16:25.586047 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.586027    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-46f024-5d47b86cfc-xt9sv"]
Sep 18 20:16:25.613922 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.613896    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"]
Sep 18 20:16:25.636541 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.636516    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-46f024-c7c9dbc54-kwk2v"]
Sep 18 20:16:25.669880 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.669857    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:25.982596 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:25.982563    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:26.199781 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:26.199688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:26.318848 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:26.318806    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5789ca3-7407-4734-8bc3-9a3d48d6db75" path="/var/lib/kubelet/pods/e5789ca3-7407-4734-8bc3-9a3d48d6db75/volumes"
Sep 18 20:16:26.319303 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:26.319281    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f16aede4-e2b4-4466-82c7-f32d2e7b5faa" path="/var/lib/kubelet/pods/f16aede4-e2b4-4466-82c7-f32d2e7b5faa/volumes"
Sep 18 20:16:27.453608 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:27.453572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:28.537568 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.537516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:28.628049 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.628011    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd" podUID="f53e2d41-22ef-41a0-b613-e2c836fbc407" containerName="main" containerID="cri-o://9a9eaa23c75e86e73321f566f266f1fbc6fede8aeca3cfbbbe67dcbe9d377da2" gracePeriod=45
Sep 18 20:16:28.875901 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.875879    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:28.977678 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.977649    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-model-cache\") pod \"f53e2d41-22ef-41a0-b613-e2c836fbc407\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") "
Sep 18 20:16:28.977678 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.977681    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-tmp-dir\") pod \"f53e2d41-22ef-41a0-b613-e2c836fbc407\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") "
Sep 18 20:16:28.977920 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.977723    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gttk8\" (UniqueName: \"kubernetes.io/projected/f53e2d41-22ef-41a0-b613-e2c836fbc407-kube-api-access-gttk8\") pod \"f53e2d41-22ef-41a0-b613-e2c836fbc407\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") "
Sep 18 20:16:28.977920 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.977777    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-dshm\") pod \"f53e2d41-22ef-41a0-b613-e2c836fbc407\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") "
Sep 18 20:16:28.977920 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.977793    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-home\") pod \"f53e2d41-22ef-41a0-b613-e2c836fbc407\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") "
Sep 18 20:16:28.977920 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.977834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53e2d41-22ef-41a0-b613-e2c836fbc407-tls-certs\") pod \"f53e2d41-22ef-41a0-b613-e2c836fbc407\" (UID: \"f53e2d41-22ef-41a0-b613-e2c836fbc407\") "
Sep 18 20:16:28.978165 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.977980    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-model-cache" (OuterVolumeSpecName: "model-cache") pod "f53e2d41-22ef-41a0-b613-e2c836fbc407" (UID: "f53e2d41-22ef-41a0-b613-e2c836fbc407"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:28.978165 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.978014    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f53e2d41-22ef-41a0-b613-e2c836fbc407" (UID: "f53e2d41-22ef-41a0-b613-e2c836fbc407"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:28.978165 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.978112    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-home" (OuterVolumeSpecName: "home") pod "f53e2d41-22ef-41a0-b613-e2c836fbc407" (UID: "f53e2d41-22ef-41a0-b613-e2c836fbc407"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:28.978165 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.978161    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-model-cache\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:16:28.978307 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.978205    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-tmp-dir\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:16:28.980183 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.980156    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f53e2d41-22ef-41a0-b613-e2c836fbc407-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f53e2d41-22ef-41a0-b613-e2c836fbc407" (UID: "f53e2d41-22ef-41a0-b613-e2c836fbc407"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:16:28.980378 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.980362    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53e2d41-22ef-41a0-b613-e2c836fbc407-kube-api-access-gttk8" (OuterVolumeSpecName: "kube-api-access-gttk8") pod "f53e2d41-22ef-41a0-b613-e2c836fbc407" (UID: "f53e2d41-22ef-41a0-b613-e2c836fbc407"). InnerVolumeSpecName "kube-api-access-gttk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:28.980442 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:28.980392    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-dshm" (OuterVolumeSpecName: "dshm") pod "f53e2d41-22ef-41a0-b613-e2c836fbc407" (UID: "f53e2d41-22ef-41a0-b613-e2c836fbc407"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:29.078999 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.078909    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gttk8\" (UniqueName: \"kubernetes.io/projected/f53e2d41-22ef-41a0-b613-e2c836fbc407-kube-api-access-gttk8\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:16:29.078999 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.078957    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-dshm\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:16:29.078999 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.078967    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53e2d41-22ef-41a0-b613-e2c836fbc407-home\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:16:29.078999 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.078975    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53e2d41-22ef-41a0-b613-e2c836fbc407-tls-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:16:29.547765 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.547730    2578 generic.go:356] "Generic (PLEG): container finished" podID="f53e2d41-22ef-41a0-b613-e2c836fbc407" containerID="9a9eaa23c75e86e73321f566f266f1fbc6fede8aeca3cfbbbe67dcbe9d377da2" exitCode=0
Sep 18 20:16:29.547765 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.547773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd" event={"ID":"f53e2d41-22ef-41a0-b613-e2c836fbc407","Type":"ContainerDied","Data":"9a9eaa23c75e86e73321f566f266f1fbc6fede8aeca3cfbbbe67dcbe9d377da2"}
Sep 18 20:16:29.548329 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.547804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd" event={"ID":"f53e2d41-22ef-41a0-b613-e2c836fbc407","Type":"ContainerDied","Data":"16da380b205af1f84ccb75ae80d0db56255f9cdc5761a39ca0087b584b68d332"}
Sep 18 20:16:29.548329 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.547825    2578 scope.go:117] "RemoveContainer" containerID="9a9eaa23c75e86e73321f566f266f1fbc6fede8aeca3cfbbbe67dcbe9d377da2"
Sep 18 20:16:29.548329 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.547774    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"
Sep 18 20:16:29.556869 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.556848    2578 scope.go:117] "RemoveContainer" containerID="9a9eaa23c75e86e73321f566f266f1fbc6fede8aeca3cfbbbe67dcbe9d377da2"
Sep 18 20:16:29.557208 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:16:29.557187    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9eaa23c75e86e73321f566f266f1fbc6fede8aeca3cfbbbe67dcbe9d377da2\": container with ID starting with 9a9eaa23c75e86e73321f566f266f1fbc6fede8aeca3cfbbbe67dcbe9d377da2 not found: ID does not exist" containerID="9a9eaa23c75e86e73321f566f266f1fbc6fede8aeca3cfbbbe67dcbe9d377da2"
Sep 18 20:16:29.557316 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.557216    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9eaa23c75e86e73321f566f266f1fbc6fede8aeca3cfbbbe67dcbe9d377da2"} err="failed to get container status \"9a9eaa23c75e86e73321f566f266f1fbc6fede8aeca3cfbbbe67dcbe9d377da2\": rpc error: code = NotFound desc = could not find container \"9a9eaa23c75e86e73321f566f266f1fbc6fede8aeca3cfbbbe67dcbe9d377da2\": container with ID starting with 9a9eaa23c75e86e73321f566f266f1fbc6fede8aeca3cfbbbe67dcbe9d377da2 not found: ID does not exist"
Sep 18 20:16:29.571807 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.571773    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"]
Sep 18 20:16:29.578178 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.578147    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-46f024/explicit-model-46f024-kserve-7d879f6b49-t6hqd"]
Sep 18 20:16:29.674390 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.674356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:29.993673 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:29.993628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:30.094787 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:30.094757    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:30.239541 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:30.239506    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:30.319815 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:30.319736    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f53e2d41-22ef-41a0-b613-e2c836fbc407" path="/var/lib/kubelet/pods/f53e2d41-22ef-41a0-b613-e2c836fbc407/volumes"
Sep 18 20:16:30.674381 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:30.674298    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:30.770449 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:30.770414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:34.951278 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:34.951244    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:35.116356 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:35.116322    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:35.278692 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:35.278651    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:35.811821 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:35.811782    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:36.209741 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:36.209637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:36.513910 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:36.513873    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:37.500946 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:37.500898    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:37.770441 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:37.770347    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:40.164628 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:40.164590    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:40.764952 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:40.764909    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:40.890489 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:40.890456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:41.424437 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:41.424403    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:42.662319 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:42.662287    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:43.893198 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:43.893162    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:45.243381 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.243349    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"]
Sep 18 20:16:45.243730 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.243676    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f53e2d41-22ef-41a0-b613-e2c836fbc407" containerName="main"
Sep 18 20:16:45.243730 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.243687    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f53e2d41-22ef-41a0-b613-e2c836fbc407" containerName="main"
Sep 18 20:16:45.243730 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.243695    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f16aede4-e2b4-4466-82c7-f32d2e7b5faa" containerName="payload-pre-processing"
Sep 18 20:16:45.243730 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.243701    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f16aede4-e2b4-4466-82c7-f32d2e7b5faa" containerName="payload-pre-processing"
Sep 18 20:16:45.243730 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.243721    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5789ca3-7407-4734-8bc3-9a3d48d6db75" containerName="maas-api"
Sep 18 20:16:45.243730 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.243726    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5789ca3-7407-4734-8bc3-9a3d48d6db75" containerName="maas-api"
Sep 18 20:16:45.243956 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.243783    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f53e2d41-22ef-41a0-b613-e2c836fbc407" containerName="main"
Sep 18 20:16:45.243956 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.243791    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f16aede4-e2b4-4466-82c7-f32d2e7b5faa" containerName="payload-pre-processing"
Sep 18 20:16:45.243956 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.243798    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5789ca3-7407-4734-8bc3-9a3d48d6db75" containerName="maas-api"
Sep 18 20:16:45.248084 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.248066    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:45.250948 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.250916    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-aba3750c-serving-cert\""
Sep 18 20:16:45.274017 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.273990    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"]
Sep 18 20:16:45.323183 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.323157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b34d51da-258c-4655-ac3d-f51e77c3b999-openshift-service-ca\") pod \"maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:45.323183 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.323186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-metrics-certs\") pod \"maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:45.323403 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.323219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-maas-api-tls\") pod \"maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:45.323403 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.323356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8xz\" (UniqueName: \"kubernetes.io/projected/b34d51da-258c-4655-ac3d-f51e77c3b999-kube-api-access-lv8xz\") pod \"maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:45.324617 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.324587    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:45.424595 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.424561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-maas-api-tls\") pod \"maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:45.424783 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.424620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8xz\" (UniqueName: \"kubernetes.io/projected/b34d51da-258c-4655-ac3d-f51e77c3b999-kube-api-access-lv8xz\") pod \"maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:45.424783 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.424659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b34d51da-258c-4655-ac3d-f51e77c3b999-openshift-service-ca\") pod \"maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:45.424783 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.424674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-metrics-certs\") pod \"maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:45.424783 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:16:45.424707    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-serving-cert: secret "maas-api-e2e-mt-aba3750c-serving-cert" not found
Sep 18 20:16:45.424783 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:16:45.424779    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-maas-api-tls podName:b34d51da-258c-4655-ac3d-f51e77c3b999 nodeName:}" failed. No retries permitted until 2026-09-18 20:16:45.924759991 +0000 UTC m=+1207.763223921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-maas-api-tls") pod "maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc" (UID: "b34d51da-258c-4655-ac3d-f51e77c3b999") : secret "maas-api-e2e-mt-aba3750c-serving-cert" not found
Sep 18 20:16:45.425405 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.425382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b34d51da-258c-4655-ac3d-f51e77c3b999-openshift-service-ca\") pod \"maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:45.427257 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.427238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-metrics-certs\") pod \"maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:45.445801 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.445780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8xz\" (UniqueName: \"kubernetes.io/projected/b34d51da-258c-4655-ac3d-f51e77c3b999-kube-api-access-lv8xz\") pod \"maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:45.644176 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.644054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:45.918484 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.918391    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:45.929522 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.929488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-maas-api-tls\") pod \"maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:45.931866 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:45.931839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-maas-api-tls\") pod \"maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:46.157729 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:46.157690    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:46.614537 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:16:46.614504    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34d51da_258c_4655_ac3d_f51e77c3b999.slice/crio-4a4a3a927111529d9f0f6d7f09f9d5893b74244da4667cfdff07f007c153ae56 WatchSource:0}: Error finding container 4a4a3a927111529d9f0f6d7f09f9d5893b74244da4667cfdff07f007c153ae56: Status 404 returned error can't find the container with id 4a4a3a927111529d9f0f6d7f09f9d5893b74244da4667cfdff07f007c153ae56
Sep 18 20:16:46.622106 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:46.622082    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"]
Sep 18 20:16:46.634828 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:46.634800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc" event={"ID":"b34d51da-258c-4655-ac3d-f51e77c3b999","Type":"ContainerStarted","Data":"4a4a3a927111529d9f0f6d7f09f9d5893b74244da4667cfdff07f007c153ae56"}
Sep 18 20:16:47.125746 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.125713    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:47.261160 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.261078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:47.376078 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.376046    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"]
Sep 18 20:16:47.381680 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.381645    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"
Sep 18 20:16:47.385617 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.385591    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-aba3750c\""
Sep 18 20:16:47.387289 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.387271    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-aba3750c-dockercfg-bxwwp\""
Sep 18 20:16:47.397190 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.397163    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"]
Sep 18 20:16:47.444087 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.444050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d129e4f-cb0e-44e2-ac63-59579e5ef811-plugins-config-volume\") pod \"payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd\" (UID: \"5d129e4f-cb0e-44e2-ac63-59579e5ef811\") " pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"
Sep 18 20:16:47.444453 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.444427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m484k\" (UniqueName: \"kubernetes.io/projected/5d129e4f-cb0e-44e2-ac63-59579e5ef811-kube-api-access-m484k\") pod \"payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd\" (UID: \"5d129e4f-cb0e-44e2-ac63-59579e5ef811\") " pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"
Sep 18 20:16:47.545539 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.545440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m484k\" (UniqueName: \"kubernetes.io/projected/5d129e4f-cb0e-44e2-ac63-59579e5ef811-kube-api-access-m484k\") pod \"payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd\" (UID: \"5d129e4f-cb0e-44e2-ac63-59579e5ef811\") " pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"
Sep 18 20:16:47.545539 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.545531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d129e4f-cb0e-44e2-ac63-59579e5ef811-plugins-config-volume\") pod \"payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd\" (UID: \"5d129e4f-cb0e-44e2-ac63-59579e5ef811\") " pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"
Sep 18 20:16:47.546243 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.546222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d129e4f-cb0e-44e2-ac63-59579e5ef811-plugins-config-volume\") pod \"payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd\" (UID: \"5d129e4f-cb0e-44e2-ac63-59579e5ef811\") " pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"
Sep 18 20:16:47.576769 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.576743    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m484k\" (UniqueName: \"kubernetes.io/projected/5d129e4f-cb0e-44e2-ac63-59579e5ef811-kube-api-access-m484k\") pod \"payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd\" (UID: \"5d129e4f-cb0e-44e2-ac63-59579e5ef811\") " pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"
Sep 18 20:16:47.641705 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.641665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc" event={"ID":"b34d51da-258c-4655-ac3d-f51e77c3b999","Type":"ContainerStarted","Data":"e4852e6b0c145665004ec558486c4be7074f3a943c14cee5982b906a0ffff5c5"}
Sep 18 20:16:47.642090 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.641763    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:47.693148 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.693116    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"
Sep 18 20:16:47.913689 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.913602    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:47.973149 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.973105    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc" podStartSLOduration=2.607508832 podStartE2EDuration="2.973088412s" totalImagesPullingTime="365.57958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:47.681014865 +0000 UTC m=+1209.519478817" watchObservedRunningTime="2026-09-18 20:16:47.973088412 +0000 UTC m=+1209.811552421"
Sep 18 20:16:47.973307 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:47.973294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"]
Sep 18 20:16:47.975014 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:16:47.974986    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d129e4f_cb0e_44e2_ac63_59579e5ef811.slice/crio-1435d94850105a00cb40e5e38524f50add8c6ccd4231e90dc74661aa26f2250d WatchSource:0}: Error finding container 1435d94850105a00cb40e5e38524f50add8c6ccd4231e90dc74661aa26f2250d: Status 404 returned error can't find the container with id 1435d94850105a00cb40e5e38524f50add8c6ccd4231e90dc74661aa26f2250d
Sep 18 20:16:48.648485 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:48.648430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd" event={"ID":"5d129e4f-cb0e-44e2-ac63-59579e5ef811","Type":"ContainerStarted","Data":"72e39dd48a95788d79ccc3abf77243652b696dc90fb863018e15f0fa4b0f683d"}
Sep 18 20:16:48.648485 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:48.648486    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd" event={"ID":"5d129e4f-cb0e-44e2-ac63-59579e5ef811","Type":"ContainerStarted","Data":"1435d94850105a00cb40e5e38524f50add8c6ccd4231e90dc74661aa26f2250d"}
Sep 18 20:16:48.649056 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:48.648515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"
Sep 18 20:16:48.671661 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:48.671605    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd" podStartSLOduration=1.671582541 podStartE2EDuration="1.671582541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:48.668399897 +0000 UTC m=+1210.506863848" watchObservedRunningTime="2026-09-18 20:16:48.671582541 +0000 UTC m=+1210.510046495"
Sep 18 20:16:48.960539 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:48.960458    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:53.653564 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:53.653536    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:16:57.473978 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:57.473923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:59.592392 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:59.592357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:16:59.656528 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:16:59.656494    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"
Sep 18 20:17:04.374229 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:04.374190    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:10.785330 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:10.785292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:11.157199 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:11.157112    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:11.472332 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:11.472299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:11.635565 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:11.635526    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:11.948563 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:11.948477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:12.476587 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:12.476551    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:13.493653 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:13.493598    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:14.593059 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:14.593013    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:15.946592 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:15.946553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:18.568716 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.568680    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"]
Sep 18 20:17:18.572357 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.572340    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.588953 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.588921    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5284dc\"/\"distinct-w0-5284dc-kserve-self-signed-certs\""
Sep 18 20:17:18.612696 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.612664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-model-cache\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.612812 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.612708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-tmp-dir\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.612812 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.612732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9djx\" (UniqueName: \"kubernetes.io/projected/77f8cb7f-4b15-4e77-a35e-18e87901082a-kube-api-access-w9djx\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.612812 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.612762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f8cb7f-4b15-4e77-a35e-18e87901082a-tls-certs\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.612928 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.612842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-home\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.612928 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.612873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-dshm\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.617611 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.617587    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"]
Sep 18 20:17:18.713826 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.713795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-home\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.714029 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.713839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-dshm\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.714029 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.713872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-model-cache\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.714029 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.713899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-tmp-dir\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.714029 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.713926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9djx\" (UniqueName: \"kubernetes.io/projected/77f8cb7f-4b15-4e77-a35e-18e87901082a-kube-api-access-w9djx\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.714029 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.713981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f8cb7f-4b15-4e77-a35e-18e87901082a-tls-certs\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.714319 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.714244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-home\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.714319 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.714286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-model-cache\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.714420 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.714389    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-tmp-dir\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.716131 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.716108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-dshm\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.716400 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.716383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f8cb7f-4b15-4e77-a35e-18e87901082a-tls-certs\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.730560 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.730536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9djx\" (UniqueName: \"kubernetes.io/projected/77f8cb7f-4b15-4e77-a35e-18e87901082a-kube-api-access-w9djx\") pod \"distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") " pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:18.773483 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.773459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:18.881732 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:18.881641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:19.172165 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:19.171953    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"]
Sep 18 20:17:19.174743 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:17:19.174715    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77f8cb7f_4b15_4e77_a35e_18e87901082a.slice/crio-69e39548df54990cca69ac297e5434b66773af34f74478a2fc544e1d9b13d078 WatchSource:0}: Error finding container 69e39548df54990cca69ac297e5434b66773af34f74478a2fc544e1d9b13d078: Status 404 returned error can't find the container with id 69e39548df54990cca69ac297e5434b66773af34f74478a2fc544e1d9b13d078
Sep 18 20:17:19.813413 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:19.813380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8" event={"ID":"77f8cb7f-4b15-4e77-a35e-18e87901082a","Type":"ContainerStarted","Data":"7d0e4119414685269eac3ca32ad817abd735f2894ca244eda0398fbc24043ebd"}
Sep 18 20:17:19.813413 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:19.813422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8" event={"ID":"77f8cb7f-4b15-4e77-a35e-18e87901082a","Type":"ContainerStarted","Data":"69e39548df54990cca69ac297e5434b66773af34f74478a2fc544e1d9b13d078"}
Sep 18 20:17:20.902239 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:20.902201    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:21.785090 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:21.785048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:21.945749 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:21.945716    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:22.258692 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:22.258656    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:22.416418 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:22.416388    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:22.572438 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:22.572346    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:22.783371 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:22.783337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:24.073126 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:24.073090    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:26.667450 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:26.667419    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:27.773473 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:27.773432    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:28.047561 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:28.047464    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:28.882490 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:28.882458    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:28.882945 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:28.882629    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:28.895114 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:28.895090    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:28.919740 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:28.919687    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8" podStartSLOduration=10.919666644 podStartE2EDuration="10.919666644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:17:19.859289634 +0000 UTC m=+1241.697753587" watchObservedRunningTime="2026-09-18 20:17:28.919666644 +0000 UTC m=+1250.758130596"
Sep 18 20:17:29.252978 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:29.252915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:29.872785 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:29.872758    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:17:30.297201 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:30.297163    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:31.187560 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:31.187527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:32.389083 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:32.389054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:33.084995 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.084960    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"]
Sep 18 20:17:33.085238 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.085212    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc" podUID="b34d51da-258c-4655-ac3d-f51e77c3b999" containerName="maas-api" containerID="cri-o://e4852e6b0c145665004ec558486c4be7074f3a943c14cee5982b906a0ffff5c5" gracePeriod=30
Sep 18 20:17:33.136584 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.136550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:33.209903 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.209871    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"]
Sep 18 20:17:33.210185 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.210155    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd" podUID="5d129e4f-cb0e-44e2-ac63-59579e5ef811" containerName="payload-processing" containerID="cri-o://72e39dd48a95788d79ccc3abf77243652b696dc90fb863018e15f0fa4b0f683d" gracePeriod=30
Sep 18 20:17:33.339175 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.339102    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:17:33.346714 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.346668    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:33.446505 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.446479    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-maas-api-tls\") pod \"b34d51da-258c-4655-ac3d-f51e77c3b999\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") "
Sep 18 20:17:33.446901 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.446533    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b34d51da-258c-4655-ac3d-f51e77c3b999-openshift-service-ca\") pod \"b34d51da-258c-4655-ac3d-f51e77c3b999\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") "
Sep 18 20:17:33.446901 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.446565    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-metrics-certs\") pod \"b34d51da-258c-4655-ac3d-f51e77c3b999\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") "
Sep 18 20:17:33.446901 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.446605    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lv8xz\" (UniqueName: \"kubernetes.io/projected/b34d51da-258c-4655-ac3d-f51e77c3b999-kube-api-access-lv8xz\") pod \"b34d51da-258c-4655-ac3d-f51e77c3b999\" (UID: \"b34d51da-258c-4655-ac3d-f51e77c3b999\") "
Sep 18 20:17:33.447132 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.446949    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34d51da-258c-4655-ac3d-f51e77c3b999-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b34d51da-258c-4655-ac3d-f51e77c3b999" (UID: "b34d51da-258c-4655-ac3d-f51e77c3b999"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:17:33.448639 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.448608    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b34d51da-258c-4655-ac3d-f51e77c3b999" (UID: "b34d51da-258c-4655-ac3d-f51e77c3b999"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:17:33.448770 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.448646    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b34d51da-258c-4655-ac3d-f51e77c3b999" (UID: "b34d51da-258c-4655-ac3d-f51e77c3b999"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:17:33.448770 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.448671    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34d51da-258c-4655-ac3d-f51e77c3b999-kube-api-access-lv8xz" (OuterVolumeSpecName: "kube-api-access-lv8xz") pod "b34d51da-258c-4655-ac3d-f51e77c3b999" (UID: "b34d51da-258c-4655-ac3d-f51e77c3b999"). InnerVolumeSpecName "kube-api-access-lv8xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:33.459560 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.459537    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"
Sep 18 20:17:33.501347 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.501314    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:33.547151 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.547110    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m484k\" (UniqueName: \"kubernetes.io/projected/5d129e4f-cb0e-44e2-ac63-59579e5ef811-kube-api-access-m484k\") pod \"5d129e4f-cb0e-44e2-ac63-59579e5ef811\" (UID: \"5d129e4f-cb0e-44e2-ac63-59579e5ef811\") "
Sep 18 20:17:33.547327 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.547190    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d129e4f-cb0e-44e2-ac63-59579e5ef811-plugins-config-volume\") pod \"5d129e4f-cb0e-44e2-ac63-59579e5ef811\" (UID: \"5d129e4f-cb0e-44e2-ac63-59579e5ef811\") "
Sep 18 20:17:33.547528 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.547505    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:17:33.547712 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.547531    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b34d51da-258c-4655-ac3d-f51e77c3b999-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:17:33.547712 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.547546    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b34d51da-258c-4655-ac3d-f51e77c3b999-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:17:33.547712 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.547559    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv8xz\" (UniqueName: \"kubernetes.io/projected/b34d51da-258c-4655-ac3d-f51e77c3b999-kube-api-access-lv8xz\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:17:33.547712 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.547586    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d129e4f-cb0e-44e2-ac63-59579e5ef811-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d129e4f-cb0e-44e2-ac63-59579e5ef811" (UID: "5d129e4f-cb0e-44e2-ac63-59579e5ef811"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:17:33.549384 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.549361    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d129e4f-cb0e-44e2-ac63-59579e5ef811-kube-api-access-m484k" (OuterVolumeSpecName: "kube-api-access-m484k") pod "5d129e4f-cb0e-44e2-ac63-59579e5ef811" (UID: "5d129e4f-cb0e-44e2-ac63-59579e5ef811"). InnerVolumeSpecName "kube-api-access-m484k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:33.606442 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.606357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:33.648160 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.648130    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m484k\" (UniqueName: \"kubernetes.io/projected/5d129e4f-cb0e-44e2-ac63-59579e5ef811-kube-api-access-m484k\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:17:33.648160 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.648157    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d129e4f-cb0e-44e2-ac63-59579e5ef811-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:17:33.883490 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.883401    2578 generic.go:356] "Generic (PLEG): container finished" podID="b34d51da-258c-4655-ac3d-f51e77c3b999" containerID="e4852e6b0c145665004ec558486c4be7074f3a943c14cee5982b906a0ffff5c5" exitCode=0
Sep 18 20:17:33.883490 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.883442    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"
Sep 18 20:17:33.883490 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.883483    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc" event={"ID":"b34d51da-258c-4655-ac3d-f51e77c3b999","Type":"ContainerDied","Data":"e4852e6b0c145665004ec558486c4be7074f3a943c14cee5982b906a0ffff5c5"}
Sep 18 20:17:33.883764 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.883531    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc" event={"ID":"b34d51da-258c-4655-ac3d-f51e77c3b999","Type":"ContainerDied","Data":"4a4a3a927111529d9f0f6d7f09f9d5893b74244da4667cfdff07f007c153ae56"}
Sep 18 20:17:33.883764 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.883544    2578 scope.go:117] "RemoveContainer" containerID="e4852e6b0c145665004ec558486c4be7074f3a943c14cee5982b906a0ffff5c5"
Sep 18 20:17:33.884874 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.884850    2578 generic.go:356] "Generic (PLEG): container finished" podID="5d129e4f-cb0e-44e2-ac63-59579e5ef811" containerID="72e39dd48a95788d79ccc3abf77243652b696dc90fb863018e15f0fa4b0f683d" exitCode=0
Sep 18 20:17:33.884990 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.884898    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"
Sep 18 20:17:33.884990 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.884949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd" event={"ID":"5d129e4f-cb0e-44e2-ac63-59579e5ef811","Type":"ContainerDied","Data":"72e39dd48a95788d79ccc3abf77243652b696dc90fb863018e15f0fa4b0f683d"}
Sep 18 20:17:33.885111 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.884990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd" event={"ID":"5d129e4f-cb0e-44e2-ac63-59579e5ef811","Type":"ContainerDied","Data":"1435d94850105a00cb40e5e38524f50add8c6ccd4231e90dc74661aa26f2250d"}
Sep 18 20:17:33.895345 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.895321    2578 scope.go:117] "RemoveContainer" containerID="e4852e6b0c145665004ec558486c4be7074f3a943c14cee5982b906a0ffff5c5"
Sep 18 20:17:33.896353 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:17:33.896319    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4852e6b0c145665004ec558486c4be7074f3a943c14cee5982b906a0ffff5c5\": container with ID starting with e4852e6b0c145665004ec558486c4be7074f3a943c14cee5982b906a0ffff5c5 not found: ID does not exist" containerID="e4852e6b0c145665004ec558486c4be7074f3a943c14cee5982b906a0ffff5c5"
Sep 18 20:17:33.896494 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.896353    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4852e6b0c145665004ec558486c4be7074f3a943c14cee5982b906a0ffff5c5"} err="failed to get container status \"e4852e6b0c145665004ec558486c4be7074f3a943c14cee5982b906a0ffff5c5\": rpc error: code = NotFound desc = could not find container \"e4852e6b0c145665004ec558486c4be7074f3a943c14cee5982b906a0ffff5c5\": container with ID starting with e4852e6b0c145665004ec558486c4be7074f3a943c14cee5982b906a0ffff5c5 not found: ID does not exist"
Sep 18 20:17:33.896494 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.896378    2578 scope.go:117] "RemoveContainer" containerID="72e39dd48a95788d79ccc3abf77243652b696dc90fb863018e15f0fa4b0f683d"
Sep 18 20:17:33.904573 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.904551    2578 scope.go:117] "RemoveContainer" containerID="72e39dd48a95788d79ccc3abf77243652b696dc90fb863018e15f0fa4b0f683d"
Sep 18 20:17:33.904849 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:17:33.904829    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72e39dd48a95788d79ccc3abf77243652b696dc90fb863018e15f0fa4b0f683d\": container with ID starting with 72e39dd48a95788d79ccc3abf77243652b696dc90fb863018e15f0fa4b0f683d not found: ID does not exist" containerID="72e39dd48a95788d79ccc3abf77243652b696dc90fb863018e15f0fa4b0f683d"
Sep 18 20:17:33.904912 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.904854    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72e39dd48a95788d79ccc3abf77243652b696dc90fb863018e15f0fa4b0f683d"} err="failed to get container status \"72e39dd48a95788d79ccc3abf77243652b696dc90fb863018e15f0fa4b0f683d\": rpc error: code = NotFound desc = could not find container \"72e39dd48a95788d79ccc3abf77243652b696dc90fb863018e15f0fa4b0f683d\": container with ID starting with 72e39dd48a95788d79ccc3abf77243652b696dc90fb863018e15f0fa4b0f683d not found: ID does not exist"
Sep 18 20:17:33.914503 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.914473    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"]
Sep 18 20:17:33.920301 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.920279    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aba3750c-7cb845fbb5-dbjgd"]
Sep 18 20:17:33.930987 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.930965    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"]
Sep 18 20:17:33.939404 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.939383    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-aba3750c-785bf96f6d-dcjsc"]
Sep 18 20:17:33.977267 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:33.977234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:34.081282 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:34.081246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:34.318096 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:34.318065    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d129e4f-cb0e-44e2-ac63-59579e5ef811" path="/var/lib/kubelet/pods/5d129e4f-cb0e-44e2-ac63-59579e5ef811/volumes"
Sep 18 20:17:34.318477 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:34.318464    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b34d51da-258c-4655-ac3d-f51e77c3b999" path="/var/lib/kubelet/pods/b34d51da-258c-4655-ac3d-f51e77c3b999/volumes"
Sep 18 20:17:35.235527 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:35.235477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:35.678610 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:35.678508    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:37.621617 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:37.621582    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:38.181099 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:38.181048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:38.361846 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:38.361782    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:39.286148 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:39.286107    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:40.936568 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:40.936532    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:41.828297 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:41.828261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:42.609236 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:42.609203    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:42.727688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:42.727647    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:42.926272 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:42.926175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:43.246180 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:43.246151    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:43.351478 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:43.351433    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:43.451744 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:43.451706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:46.378236 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.378193    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"]
Sep 18 20:17:46.378688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.378473    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx" podUID="5067c12c-2149-4dcc-a372-9956709543f9" containerName="maas-api" containerID="cri-o://89232538f6a2fedd38fbbaa147a773a256b36493047fc5b8a4afe60880a8c7de" gracePeriod=30
Sep 18 20:17:46.413848 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.413815    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:46.458340 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.458310    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"]
Sep 18 20:17:46.458553 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.458530    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww" podUID="f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70" containerName="payload-processing" containerID="cri-o://f40cf8e65487143507449755544386f4f5707b797bdd80ce727a7ce97dc74539" gracePeriod=30
Sep 18 20:17:46.501556 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.501524    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:46.600494 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.600462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:46.701430 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.701360    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:46.721918 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.721890    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"
Sep 18 20:17:46.806382 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.806353    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:46.862858 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.862830    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8qw9\" (UniqueName: \"kubernetes.io/projected/f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70-kube-api-access-k8qw9\") pod \"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70\" (UID: \"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70\") "
Sep 18 20:17:46.863011 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.862927    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70-plugins-config-volume\") pod \"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70\" (UID: \"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70\") "
Sep 18 20:17:46.863281 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.863257    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70" (UID: "f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:17:46.864918 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.864892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70-kube-api-access-k8qw9" (OuterVolumeSpecName: "kube-api-access-k8qw9") pod "f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70" (UID: "f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70"). InnerVolumeSpecName "kube-api-access-k8qw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:46.958127 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.958071    2578 generic.go:356] "Generic (PLEG): container finished" podID="f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70" containerID="f40cf8e65487143507449755544386f4f5707b797bdd80ce727a7ce97dc74539" exitCode=0
Sep 18 20:17:46.958127 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.958118    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"
Sep 18 20:17:46.958250 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.958121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww" event={"ID":"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70","Type":"ContainerDied","Data":"f40cf8e65487143507449755544386f4f5707b797bdd80ce727a7ce97dc74539"}
Sep 18 20:17:46.958291 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.958247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww" event={"ID":"f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70","Type":"ContainerDied","Data":"ee66711d6affd9ac005ccfa4c165d11909c8b16f28baf8efc191ed692320cb74"}
Sep 18 20:17:46.958291 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.958282    2578 scope.go:117] "RemoveContainer" containerID="f40cf8e65487143507449755544386f4f5707b797bdd80ce727a7ce97dc74539"
Sep 18 20:17:46.964200 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.964176    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:17:46.964293 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:46.964205    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8qw9\" (UniqueName: \"kubernetes.io/projected/f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70-kube-api-access-k8qw9\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:17:47.015513 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.015490    2578 scope.go:117] "RemoveContainer" containerID="f40cf8e65487143507449755544386f4f5707b797bdd80ce727a7ce97dc74539"
Sep 18 20:17:47.015843 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:17:47.015819    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40cf8e65487143507449755544386f4f5707b797bdd80ce727a7ce97dc74539\": container with ID starting with f40cf8e65487143507449755544386f4f5707b797bdd80ce727a7ce97dc74539 not found: ID does not exist" containerID="f40cf8e65487143507449755544386f4f5707b797bdd80ce727a7ce97dc74539"
Sep 18 20:17:47.015903 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.015853    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40cf8e65487143507449755544386f4f5707b797bdd80ce727a7ce97dc74539"} err="failed to get container status \"f40cf8e65487143507449755544386f4f5707b797bdd80ce727a7ce97dc74539\": rpc error: code = NotFound desc = could not find container \"f40cf8e65487143507449755544386f4f5707b797bdd80ce727a7ce97dc74539\": container with ID starting with f40cf8e65487143507449755544386f4f5707b797bdd80ce727a7ce97dc74539 not found: ID does not exist"
Sep 18 20:17:47.046970 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.046918    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"]
Sep 18 20:17:47.068420 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.068393    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3a8052-5cc57f4755-vdxww"]
Sep 18 20:17:47.118786 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.118762    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:17:47.265497 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.265448    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-metrics-certs\") pod \"5067c12c-2149-4dcc-a372-9956709543f9\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") "
Sep 18 20:17:47.265669 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.265520    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5067c12c-2149-4dcc-a372-9956709543f9-openshift-service-ca\") pod \"5067c12c-2149-4dcc-a372-9956709543f9\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") "
Sep 18 20:17:47.265669 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.265554    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgbl6\" (UniqueName: \"kubernetes.io/projected/5067c12c-2149-4dcc-a372-9956709543f9-kube-api-access-lgbl6\") pod \"5067c12c-2149-4dcc-a372-9956709543f9\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") "
Sep 18 20:17:47.265669 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.265573    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-maas-api-tls\") pod \"5067c12c-2149-4dcc-a372-9956709543f9\" (UID: \"5067c12c-2149-4dcc-a372-9956709543f9\") "
Sep 18 20:17:47.265968 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.265919    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5067c12c-2149-4dcc-a372-9956709543f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5067c12c-2149-4dcc-a372-9956709543f9" (UID: "5067c12c-2149-4dcc-a372-9956709543f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:17:47.267609 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.267586    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5067c12c-2149-4dcc-a372-9956709543f9-kube-api-access-lgbl6" (OuterVolumeSpecName: "kube-api-access-lgbl6") pod "5067c12c-2149-4dcc-a372-9956709543f9" (UID: "5067c12c-2149-4dcc-a372-9956709543f9"). InnerVolumeSpecName "kube-api-access-lgbl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:47.267609 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.267598    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5067c12c-2149-4dcc-a372-9956709543f9" (UID: "5067c12c-2149-4dcc-a372-9956709543f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:17:47.267729 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.267623    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5067c12c-2149-4dcc-a372-9956709543f9" (UID: "5067c12c-2149-4dcc-a372-9956709543f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:17:47.275518 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.275495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:47.366859 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.366827    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:17:47.366859 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.366856    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5067c12c-2149-4dcc-a372-9956709543f9-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:17:47.366859 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.366865    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgbl6\" (UniqueName: \"kubernetes.io/projected/5067c12c-2149-4dcc-a372-9956709543f9-kube-api-access-lgbl6\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:17:47.367108 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.366873    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5067c12c-2149-4dcc-a372-9956709543f9-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:17:47.425481 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.425445    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:47.964231 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.964191    2578 generic.go:356] "Generic (PLEG): container finished" podID="5067c12c-2149-4dcc-a372-9956709543f9" containerID="89232538f6a2fedd38fbbaa147a773a256b36493047fc5b8a4afe60880a8c7de" exitCode=0
Sep 18 20:17:47.964416 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.964250    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"
Sep 18 20:17:47.964416 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.964277    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx" event={"ID":"5067c12c-2149-4dcc-a372-9956709543f9","Type":"ContainerDied","Data":"89232538f6a2fedd38fbbaa147a773a256b36493047fc5b8a4afe60880a8c7de"}
Sep 18 20:17:47.964416 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.964360    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx" event={"ID":"5067c12c-2149-4dcc-a372-9956709543f9","Type":"ContainerDied","Data":"5cc48d622995f3bff7679181299f7fc5d5762378ce24248fe30b28c3e344d44b"}
Sep 18 20:17:47.964416 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.964388    2578 scope.go:117] "RemoveContainer" containerID="89232538f6a2fedd38fbbaa147a773a256b36493047fc5b8a4afe60880a8c7de"
Sep 18 20:17:47.973326 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.973304    2578 scope.go:117] "RemoveContainer" containerID="89232538f6a2fedd38fbbaa147a773a256b36493047fc5b8a4afe60880a8c7de"
Sep 18 20:17:47.973616 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:17:47.973598    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89232538f6a2fedd38fbbaa147a773a256b36493047fc5b8a4afe60880a8c7de\": container with ID starting with 89232538f6a2fedd38fbbaa147a773a256b36493047fc5b8a4afe60880a8c7de not found: ID does not exist" containerID="89232538f6a2fedd38fbbaa147a773a256b36493047fc5b8a4afe60880a8c7de"
Sep 18 20:17:47.973665 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:47.973625    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89232538f6a2fedd38fbbaa147a773a256b36493047fc5b8a4afe60880a8c7de"} err="failed to get container status \"89232538f6a2fedd38fbbaa147a773a256b36493047fc5b8a4afe60880a8c7de\": rpc error: code = NotFound desc = could not find container \"89232538f6a2fedd38fbbaa147a773a256b36493047fc5b8a4afe60880a8c7de\": container with ID starting with 89232538f6a2fedd38fbbaa147a773a256b36493047fc5b8a4afe60880a8c7de not found: ID does not exist"
Sep 18 20:17:48.006753 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:48.006726    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"]
Sep 18 20:17:48.012563 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:48.012541    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3a8052-b67db748c-j5ldx"]
Sep 18 20:17:48.318918 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:48.318840    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5067c12c-2149-4dcc-a372-9956709543f9" path="/var/lib/kubelet/pods/5067c12c-2149-4dcc-a372-9956709543f9/volumes"
Sep 18 20:17:48.319239 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:48.319226    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70" path="/var/lib/kubelet/pods/f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70/volumes"
Sep 18 20:17:48.802882 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:48.802847    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:50.020040 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:50.020003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:51.533530 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:51.533488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:51.690184 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:51.690149    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:51.861531 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:51.861447    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:52.016650 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:52.016619    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:54.917879 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:54.917845    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:55.022275 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:55.022245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:55.359545 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:55.359510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:55.872215 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:55.872176    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:57.109052 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:57.109024    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:58.636959 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:58.636903    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:17:58.864093 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:17:58.864056    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:00.482607 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:00.482570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:01.508648 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:01.508617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:01.597645 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:01.597608    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:02.510784 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:02.510730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:02.932038 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:02.931950    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:03.075390 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:03.075354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:07.082816 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:07.082783    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:07.417421 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:07.417325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:08.176150 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:08.176103    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:09.119308 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:09.119275    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:10.045977 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:10.045923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:11.398515 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.398483    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"]
Sep 18 20:18:11.398886 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.398833    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70" containerName="payload-processing"
Sep 18 20:18:11.398886 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.398846    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70" containerName="payload-processing"
Sep 18 20:18:11.398886 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.398853    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b34d51da-258c-4655-ac3d-f51e77c3b999" containerName="maas-api"
Sep 18 20:18:11.398886 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.398860    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b34d51da-258c-4655-ac3d-f51e77c3b999" containerName="maas-api"
Sep 18 20:18:11.398886 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.398868    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5067c12c-2149-4dcc-a372-9956709543f9" containerName="maas-api"
Sep 18 20:18:11.398886 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.398876    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5067c12c-2149-4dcc-a372-9956709543f9" containerName="maas-api"
Sep 18 20:18:11.399151 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.398898    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d129e4f-cb0e-44e2-ac63-59579e5ef811" containerName="payload-processing"
Sep 18 20:18:11.399151 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.398904    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d129e4f-cb0e-44e2-ac63-59579e5ef811" containerName="payload-processing"
Sep 18 20:18:11.399151 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.398988    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d129e4f-cb0e-44e2-ac63-59579e5ef811" containerName="payload-processing"
Sep 18 20:18:11.399151 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.398998    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b34d51da-258c-4655-ac3d-f51e77c3b999" containerName="maas-api"
Sep 18 20:18:11.399151 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.399005    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cfaa30-ace3-4d7b-bbd6-3613b7b73f70" containerName="payload-processing"
Sep 18 20:18:11.399151 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.399011    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5067c12c-2149-4dcc-a372-9956709543f9" containerName="maas-api"
Sep 18 20:18:11.401887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.401856    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:11.410435 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:18:11.410411    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" is forbidden: User \"system:node:ip-10-0-132-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-132-172.ec2.internal' and this object" podUID="7fd8789f-ad30-4250-9127-5992c719f1b1" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:11.410520 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.410485    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-5319c6-serving-cert\""
Sep 18 20:18:11.437174 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.437145    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"]
Sep 18 20:18:11.467635 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.467610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:11.467754 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.467649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd8789f-ad30-4250-9127-5992c719f1b1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:11.467754 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.467671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxcqt\" (UniqueName: \"kubernetes.io/projected/7fd8789f-ad30-4250-9127-5992c719f1b1-kube-api-access-zxcqt\") pod \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:11.467832 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.467811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:11.481783 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.481754    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:11.568373 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.568336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:11.568549 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.568381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:11.568549 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.568425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd8789f-ad30-4250-9127-5992c719f1b1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:11.568549 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.568452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxcqt\" (UniqueName: \"kubernetes.io/projected/7fd8789f-ad30-4250-9127-5992c719f1b1-kube-api-access-zxcqt\") pod \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:11.568680 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:18:11.568548    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-serving-cert: secret "maas-api-e2e-shared-a-w2-5319c6-serving-cert" not found
Sep 18 20:18:11.568680 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:18:11.568631    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-maas-api-tls podName:7fd8789f-ad30-4250-9127-5992c719f1b1 nodeName:}" failed. No retries permitted until 2026-09-18 20:18:12.068612584 +0000 UTC m=+1293.907076526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-maas-api-tls") pod "maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj" (UID: "7fd8789f-ad30-4250-9127-5992c719f1b1") : secret "maas-api-e2e-shared-a-w2-5319c6-serving-cert" not found
Sep 18 20:18:11.569203 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.569181    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd8789f-ad30-4250-9127-5992c719f1b1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:11.570860 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.570844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:11.585235 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.585209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxcqt\" (UniqueName: \"kubernetes.io/projected/7fd8789f-ad30-4250-9127-5992c719f1b1-kube-api-access-zxcqt\") pod \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:11.616793 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.616754    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:11.950323 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:11.950287    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:12.074059 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.074016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:12.076660 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.076636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:12.257850 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.257808    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:12.312535 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.312495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:12.415530 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.415489    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:12.524134 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.524047    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"]
Sep 18 20:18:12.527623 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.527600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"
Sep 18 20:18:12.532627 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.532607    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5319c6\""
Sep 18 20:18:12.556559 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.556532    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"]
Sep 18 20:18:12.579084 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.579049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7769618-6fa3-4145-9912-4ae09261182a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm\" (UID: \"a7769618-6fa3-4145-9912-4ae09261182a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"
Sep 18 20:18:12.579226 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.579135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt26p\" (UniqueName: \"kubernetes.io/projected/a7769618-6fa3-4145-9912-4ae09261182a-kube-api-access-lt26p\") pod \"payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm\" (UID: \"a7769618-6fa3-4145-9912-4ae09261182a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"
Sep 18 20:18:12.610291 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.610264    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"]
Sep 18 20:18:12.612329 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:18:12.612304    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd8789f_ad30_4250_9127_5992c719f1b1.slice/crio-12ec601f402583b44e8aa8246813a0bcb82e948a496a60b7fb6d890d653d763a WatchSource:0}: Error finding container 12ec601f402583b44e8aa8246813a0bcb82e948a496a60b7fb6d890d653d763a: Status 404 returned error can't find the container with id 12ec601f402583b44e8aa8246813a0bcb82e948a496a60b7fb6d890d653d763a
Sep 18 20:18:12.679959 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.679898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7769618-6fa3-4145-9912-4ae09261182a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm\" (UID: \"a7769618-6fa3-4145-9912-4ae09261182a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"
Sep 18 20:18:12.680163 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.679999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt26p\" (UniqueName: \"kubernetes.io/projected/a7769618-6fa3-4145-9912-4ae09261182a-kube-api-access-lt26p\") pod \"payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm\" (UID: \"a7769618-6fa3-4145-9912-4ae09261182a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"
Sep 18 20:18:12.680948 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.680909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7769618-6fa3-4145-9912-4ae09261182a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm\" (UID: \"a7769618-6fa3-4145-9912-4ae09261182a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"
Sep 18 20:18:12.693838 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.693814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt26p\" (UniqueName: \"kubernetes.io/projected/a7769618-6fa3-4145-9912-4ae09261182a-kube-api-access-lt26p\") pod \"payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm\" (UID: \"a7769618-6fa3-4145-9912-4ae09261182a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"
Sep 18 20:18:12.838897 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.838814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"
Sep 18 20:18:12.986085 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:12.986039    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:13.058501 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.058468    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"]
Sep 18 20:18:13.058708 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.058686    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp" podUID="390d9f6b-73ee-480b-b514-b4a48a09a31e" containerName="payload-pre-processing" containerID="cri-o://d801ab16f1f31fdb06232fa092565a8ebd32c738913295757698078c3ee0a4d1" gracePeriod=30
Sep 18 20:18:13.083678 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.083653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj" event={"ID":"7fd8789f-ad30-4250-9127-5992c719f1b1","Type":"ContainerStarted","Data":"12ec601f402583b44e8aa8246813a0bcb82e948a496a60b7fb6d890d653d763a"}
Sep 18 20:18:13.143380 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.143349    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"]
Sep 18 20:18:13.143986 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:18:13.143965    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7769618_6fa3_4145_9912_4ae09261182a.slice/crio-65ca2862d1584b03dc4d027bf21655bcc056340ff40bf9cd5fdca10390ccf716 WatchSource:0}: Error finding container 65ca2862d1584b03dc4d027bf21655bcc056340ff40bf9cd5fdca10390ccf716: Status 404 returned error can't find the container with id 65ca2862d1584b03dc4d027bf21655bcc056340ff40bf9cd5fdca10390ccf716
Sep 18 20:18:13.212657 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.212468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:13.304353 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.304329    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"
Sep 18 20:18:13.385417 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.385389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390d9f6b-73ee-480b-b514-b4a48a09a31e-plugins-config-volume\") pod \"390d9f6b-73ee-480b-b514-b4a48a09a31e\" (UID: \"390d9f6b-73ee-480b-b514-b4a48a09a31e\") "
Sep 18 20:18:13.385562 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.385455    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4hll\" (UniqueName: \"kubernetes.io/projected/390d9f6b-73ee-480b-b514-b4a48a09a31e-kube-api-access-h4hll\") pod \"390d9f6b-73ee-480b-b514-b4a48a09a31e\" (UID: \"390d9f6b-73ee-480b-b514-b4a48a09a31e\") "
Sep 18 20:18:13.385761 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.385738    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390d9f6b-73ee-480b-b514-b4a48a09a31e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "390d9f6b-73ee-480b-b514-b4a48a09a31e" (UID: "390d9f6b-73ee-480b-b514-b4a48a09a31e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:18:13.387467 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.387440    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390d9f6b-73ee-480b-b514-b4a48a09a31e-kube-api-access-h4hll" (OuterVolumeSpecName: "kube-api-access-h4hll") pod "390d9f6b-73ee-480b-b514-b4a48a09a31e" (UID: "390d9f6b-73ee-480b-b514-b4a48a09a31e"). InnerVolumeSpecName "kube-api-access-h4hll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:18:13.468654 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.468613    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:13.492434 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.492401    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/390d9f6b-73ee-480b-b514-b4a48a09a31e-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:18:13.492434 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.492438    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4hll\" (UniqueName: \"kubernetes.io/projected/390d9f6b-73ee-480b-b514-b4a48a09a31e-kube-api-access-h4hll\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:18:13.764394 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.764302    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:13.946903 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:13.946867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:14.090922 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.090832    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj" event={"ID":"7fd8789f-ad30-4250-9127-5992c719f1b1","Type":"ContainerStarted","Data":"abf14a1a8d6e96f9b87643accfcdd9654277cbb6b06c64ad7194bbac95b138cf"}
Sep 18 20:18:14.091113 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.090962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:14.092590 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.092559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm" event={"ID":"a7769618-6fa3-4145-9912-4ae09261182a","Type":"ContainerStarted","Data":"20b56923672b0e534d0087f34b3153e1f9770ed294a081a601efe44fd5eff820"}
Sep 18 20:18:14.092707 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.092616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm" event={"ID":"a7769618-6fa3-4145-9912-4ae09261182a","Type":"ContainerStarted","Data":"65ca2862d1584b03dc4d027bf21655bcc056340ff40bf9cd5fdca10390ccf716"}
Sep 18 20:18:14.092707 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.092647    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"
Sep 18 20:18:14.093857 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.093834    2578 generic.go:356] "Generic (PLEG): container finished" podID="390d9f6b-73ee-480b-b514-b4a48a09a31e" containerID="d801ab16f1f31fdb06232fa092565a8ebd32c738913295757698078c3ee0a4d1" exitCode=0
Sep 18 20:18:14.093953 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.093870    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"
Sep 18 20:18:14.093953 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.093903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp" event={"ID":"390d9f6b-73ee-480b-b514-b4a48a09a31e","Type":"ContainerDied","Data":"d801ab16f1f31fdb06232fa092565a8ebd32c738913295757698078c3ee0a4d1"}
Sep 18 20:18:14.094036 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.093957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp" event={"ID":"390d9f6b-73ee-480b-b514-b4a48a09a31e","Type":"ContainerDied","Data":"d3ebfa3f439ae5f37afe46a05c4dae139a8c7c7376653583f57daa64054372b0"}
Sep 18 20:18:14.094036 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.093980    2578 scope.go:117] "RemoveContainer" containerID="d801ab16f1f31fdb06232fa092565a8ebd32c738913295757698078c3ee0a4d1"
Sep 18 20:18:14.103083 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.103064    2578 scope.go:117] "RemoveContainer" containerID="d801ab16f1f31fdb06232fa092565a8ebd32c738913295757698078c3ee0a4d1"
Sep 18 20:18:14.103358 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:18:14.103335    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d801ab16f1f31fdb06232fa092565a8ebd32c738913295757698078c3ee0a4d1\": container with ID starting with d801ab16f1f31fdb06232fa092565a8ebd32c738913295757698078c3ee0a4d1 not found: ID does not exist" containerID="d801ab16f1f31fdb06232fa092565a8ebd32c738913295757698078c3ee0a4d1"
Sep 18 20:18:14.103464 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.103363    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d801ab16f1f31fdb06232fa092565a8ebd32c738913295757698078c3ee0a4d1"} err="failed to get container status \"d801ab16f1f31fdb06232fa092565a8ebd32c738913295757698078c3ee0a4d1\": rpc error: code = NotFound desc = could not find container \"d801ab16f1f31fdb06232fa092565a8ebd32c738913295757698078c3ee0a4d1\": container with ID starting with d801ab16f1f31fdb06232fa092565a8ebd32c738913295757698078c3ee0a4d1 not found: ID does not exist"
Sep 18 20:18:14.119149 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.119121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:14.121604 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.121562    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj" podStartSLOduration=2.672253618 podStartE2EDuration="3.121545581s" totalImagesPullingTime="449.291963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:14.121351187 +0000 UTC m=+1295.959815153" watchObservedRunningTime="2026-09-18 20:18:14.121545581 +0000 UTC m=+1295.960009535"
Sep 18 20:18:14.179123 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.179096    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"]
Sep 18 20:18:14.184685 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.184658    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c34fad-b4565dc4b-qzggp"]
Sep 18 20:18:14.318083 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:14.318046    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="390d9f6b-73ee-480b-b514-b4a48a09a31e" path="/var/lib/kubelet/pods/390d9f6b-73ee-480b-b514-b4a48a09a31e/volumes"
Sep 18 20:18:15.784750 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:15.784715    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:17.304838 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:17.304796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:18.855189 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:18.855155    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:20.113210 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:20.113178    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:18:20.157770 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:20.157724    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm" podStartSLOduration=8.157708357 podStartE2EDuration="8.157708357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:14.21101139 +0000 UTC m=+1296.049475342" watchObservedRunningTime="2026-09-18 20:18:20.157708357 +0000 UTC m=+1301.996172309"
Sep 18 20:18:20.581255 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:20.581220    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:22.597794 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:22.597759    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:23.991877 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:23.991845    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:24.193641 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:24.193606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:24.633448 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:24.633412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:25.109322 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:25.109294    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"
Sep 18 20:18:25.219610 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:25.219569    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:26.156675 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:26.156641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:26.358068 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:26.358024    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:26.734762 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:26.734724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:26.963155 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:26.963122    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:27.290943 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:27.290910    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:27.613652 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:27.613555    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:28.001461 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:28.001430    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:28.203805 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:28.203771    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:29.555151 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:29.555107    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:30.166815 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:30.166782    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:30.338914 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:30.338882    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:31.066600 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:31.066567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:31.170302 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:31.170274    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:31.410080 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:31.409989    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:32.697357 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:32.697326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:33.978368 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:33.978329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:34.077085 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.077046    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:34.313673 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.313588    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:34.547665 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.547634    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:34.657154 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.657056    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:34.762921 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.762886    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"]
Sep 18 20:18:34.763272 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.763257    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="390d9f6b-73ee-480b-b514-b4a48a09a31e" containerName="payload-pre-processing"
Sep 18 20:18:34.763316 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.763274    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="390d9f6b-73ee-480b-b514-b4a48a09a31e" containerName="payload-pre-processing"
Sep 18 20:18:34.763355 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.763338    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="390d9f6b-73ee-480b-b514-b4a48a09a31e" containerName="payload-pre-processing"
Sep 18 20:18:34.765818 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.765799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"
Sep 18 20:18:34.768348 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.768320    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c5206e-dockercfg-wt8fq\""
Sep 18 20:18:34.768484 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.768382    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c5206e\""
Sep 18 20:18:34.779115 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.779090    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"]
Sep 18 20:18:34.880546 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.880513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbjww\" (UniqueName: \"kubernetes.io/projected/bae2c0d9-df61-4a95-a80e-d375a75f9722-kube-api-access-dbjww\") pod \"payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t\" (UID: \"bae2c0d9-df61-4a95-a80e-d375a75f9722\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"
Sep 18 20:18:34.880712 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.880566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae2c0d9-df61-4a95-a80e-d375a75f9722-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t\" (UID: \"bae2c0d9-df61-4a95-a80e-d375a75f9722\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"
Sep 18 20:18:34.981516 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.981487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbjww\" (UniqueName: \"kubernetes.io/projected/bae2c0d9-df61-4a95-a80e-d375a75f9722-kube-api-access-dbjww\") pod \"payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t\" (UID: \"bae2c0d9-df61-4a95-a80e-d375a75f9722\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"
Sep 18 20:18:34.981879 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.981537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae2c0d9-df61-4a95-a80e-d375a75f9722-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t\" (UID: \"bae2c0d9-df61-4a95-a80e-d375a75f9722\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"
Sep 18 20:18:34.982232 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.982211    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae2c0d9-df61-4a95-a80e-d375a75f9722-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t\" (UID: \"bae2c0d9-df61-4a95-a80e-d375a75f9722\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"
Sep 18 20:18:34.992148 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:34.992128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbjww\" (UniqueName: \"kubernetes.io/projected/bae2c0d9-df61-4a95-a80e-d375a75f9722-kube-api-access-dbjww\") pod \"payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t\" (UID: \"bae2c0d9-df61-4a95-a80e-d375a75f9722\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"
Sep 18 20:18:35.024538 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.024507    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"]
Sep 18 20:18:35.027973 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.027957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.031064 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.031046    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-40a1a5-serving-cert\""
Sep 18 20:18:35.042644 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.042617    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"]
Sep 18 20:18:35.076032 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.076000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"
Sep 18 20:18:35.125013 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.124978    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:35.183411 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.183367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.183619 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.183426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwtn\" (UniqueName: \"kubernetes.io/projected/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-kube-api-access-hnwtn\") pod \"maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.183619 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.183529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.183619 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.183585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.234910 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.234833    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:35.285052 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.285013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.285249 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.285076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwtn\" (UniqueName: \"kubernetes.io/projected/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-kube-api-access-hnwtn\") pod \"maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.285249 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.285124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.285249 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.285171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.285849 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.285824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.287704 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.287679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.287822 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.287781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.298873 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.298849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwtn\" (UniqueName: \"kubernetes.io/projected/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-kube-api-access-hnwtn\") pod \"maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.338433 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.338400    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:35.340329 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.340303    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:35.349357 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.349337    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"]
Sep 18 20:18:35.351256 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:18:35.351179    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae2c0d9_df61_4a95_a80e_d375a75f9722.slice/crio-0b253fa2dad633d3bffc674c5c566cb013c62d26a1985311a058544ab45040ec WatchSource:0}: Error finding container 0b253fa2dad633d3bffc674c5c566cb013c62d26a1985311a058544ab45040ec: Status 404 returned error can't find the container with id 0b253fa2dad633d3bffc674c5c566cb013c62d26a1985311a058544ab45040ec
Sep 18 20:18:35.623124 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.623047    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"]
Sep 18 20:18:35.625838 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:18:35.625809    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12d929b_4cdd_4aeb_bc3e_b23dcce48b77.slice/crio-54d48b637b79884b0c19318b4b4e41e93ae1d29275d2c8e1dcc1970bfba19561 WatchSource:0}: Error finding container 54d48b637b79884b0c19318b4b4e41e93ae1d29275d2c8e1dcc1970bfba19561: Status 404 returned error can't find the container with id 54d48b637b79884b0c19318b4b4e41e93ae1d29275d2c8e1dcc1970bfba19561
Sep 18 20:18:35.837962 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.837909    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:35.934489 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:35.934460    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:36.038215 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.038184    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"]
Sep 18 20:18:36.041526 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.041507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"
Sep 18 20:18:36.045822 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.045790    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-40a1a5\""
Sep 18 20:18:36.045822 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.045817    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-40a1a5-dockercfg-mmtqj\""
Sep 18 20:18:36.053352 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.053327    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"]
Sep 18 20:18:36.127279 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.127247    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"]
Sep 18 20:18:36.131236 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.131213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"
Sep 18 20:18:36.155551 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.155479    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"]
Sep 18 20:18:36.193381 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.193306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62918655-ac3d-4e65-b7d1-84a1b797f5a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn\" (UID: \"62918655-ac3d-4e65-b7d1-84a1b797f5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"
Sep 18 20:18:36.193381 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.193341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbhfn\" (UniqueName: \"kubernetes.io/projected/62918655-ac3d-4e65-b7d1-84a1b797f5a9-kube-api-access-xbhfn\") pod \"payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn\" (UID: \"62918655-ac3d-4e65-b7d1-84a1b797f5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"
Sep 18 20:18:36.216653 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.216625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch" event={"ID":"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77","Type":"ContainerStarted","Data":"2ed227f3a53c507daa0aec96011a5bdc82f95f97ab9e553f844ff062e7d573b2"}
Sep 18 20:18:36.216813 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.216677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch" event={"ID":"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77","Type":"ContainerStarted","Data":"54d48b637b79884b0c19318b4b4e41e93ae1d29275d2c8e1dcc1970bfba19561"}
Sep 18 20:18:36.216813 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.216701    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:36.218188 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.218166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t" event={"ID":"bae2c0d9-df61-4a95-a80e-d375a75f9722","Type":"ContainerStarted","Data":"1f22421567a5f160a71c0747d5d05f4260b0c23f0747282b1a5c9ac6e9b7d631"}
Sep 18 20:18:36.218324 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.218202    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t" event={"ID":"bae2c0d9-df61-4a95-a80e-d375a75f9722","Type":"ContainerStarted","Data":"0b253fa2dad633d3bffc674c5c566cb013c62d26a1985311a058544ab45040ec"}
Sep 18 20:18:36.218324 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.218239    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"
Sep 18 20:18:36.254577 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.254531    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch" podStartSLOduration=0.959311037 podStartE2EDuration="1.254517954s" totalImagesPullingTime="295.206917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:36.252957361 +0000 UTC m=+1318.091421313" watchObservedRunningTime="2026-09-18 20:18:36.254517954 +0000 UTC m=+1318.092981906"
Sep 18 20:18:36.285645 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.285589    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t" podStartSLOduration=2.285572166 podStartE2EDuration="2.285572166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:36.284244075 +0000 UTC m=+1318.122708028" watchObservedRunningTime="2026-09-18 20:18:36.285572166 +0000 UTC m=+1318.124036123"
Sep 18 20:18:36.294657 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.294639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62918655-ac3d-4e65-b7d1-84a1b797f5a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn\" (UID: \"62918655-ac3d-4e65-b7d1-84a1b797f5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"
Sep 18 20:18:36.294731 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.294679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbhfn\" (UniqueName: \"kubernetes.io/projected/62918655-ac3d-4e65-b7d1-84a1b797f5a9-kube-api-access-xbhfn\") pod \"payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn\" (UID: \"62918655-ac3d-4e65-b7d1-84a1b797f5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"
Sep 18 20:18:36.294795 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.294731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzznh\" (UniqueName: \"kubernetes.io/projected/440a5a06-5f73-40f5-b7d3-90b9f6ae925a-kube-api-access-vzznh\") pod \"payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc\" (UID: \"440a5a06-5f73-40f5-b7d3-90b9f6ae925a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"
Sep 18 20:18:36.294849 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.294798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440a5a06-5f73-40f5-b7d3-90b9f6ae925a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc\" (UID: \"440a5a06-5f73-40f5-b7d3-90b9f6ae925a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"
Sep 18 20:18:36.295406 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.295388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62918655-ac3d-4e65-b7d1-84a1b797f5a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn\" (UID: \"62918655-ac3d-4e65-b7d1-84a1b797f5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"
Sep 18 20:18:36.317965 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.317924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbhfn\" (UniqueName: \"kubernetes.io/projected/62918655-ac3d-4e65-b7d1-84a1b797f5a9-kube-api-access-xbhfn\") pod \"payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn\" (UID: \"62918655-ac3d-4e65-b7d1-84a1b797f5a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"
Sep 18 20:18:36.352616 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.352583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"
Sep 18 20:18:36.396299 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.396253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzznh\" (UniqueName: \"kubernetes.io/projected/440a5a06-5f73-40f5-b7d3-90b9f6ae925a-kube-api-access-vzznh\") pod \"payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc\" (UID: \"440a5a06-5f73-40f5-b7d3-90b9f6ae925a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"
Sep 18 20:18:36.396299 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.396307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440a5a06-5f73-40f5-b7d3-90b9f6ae925a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc\" (UID: \"440a5a06-5f73-40f5-b7d3-90b9f6ae925a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"
Sep 18 20:18:36.397091 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.397069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440a5a06-5f73-40f5-b7d3-90b9f6ae925a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc\" (UID: \"440a5a06-5f73-40f5-b7d3-90b9f6ae925a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"
Sep 18 20:18:36.411379 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.409827    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzznh\" (UniqueName: \"kubernetes.io/projected/440a5a06-5f73-40f5-b7d3-90b9f6ae925a-kube-api-access-vzznh\") pod \"payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc\" (UID: \"440a5a06-5f73-40f5-b7d3-90b9f6ae925a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"
Sep 18 20:18:36.443481 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.443404    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"
Sep 18 20:18:36.460218 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.460189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:36.637844 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.637819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"]
Sep 18 20:18:36.639623 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:18:36.639594    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62918655_ac3d_4e65_b7d1_84a1b797f5a9.slice/crio-9a7615539b31b9d91a23e2b6353157c7be94ee8a277ace8d39de6d35ca93c27c WatchSource:0}: Error finding container 9a7615539b31b9d91a23e2b6353157c7be94ee8a277ace8d39de6d35ca93c27c: Status 404 returned error can't find the container with id 9a7615539b31b9d91a23e2b6353157c7be94ee8a277ace8d39de6d35ca93c27c
Sep 18 20:18:36.734234 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:18:36.734206    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440a5a06_5f73_40f5_b7d3_90b9f6ae925a.slice/crio-0bdaca510af7dfa411fb33102c1ab2677d08e6cb4cc1b8c21d05ae3a0af269e7 WatchSource:0}: Error finding container 0bdaca510af7dfa411fb33102c1ab2677d08e6cb4cc1b8c21d05ae3a0af269e7: Status 404 returned error can't find the container with id 0bdaca510af7dfa411fb33102c1ab2677d08e6cb4cc1b8c21d05ae3a0af269e7
Sep 18 20:18:36.736356 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:36.736333    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"]
Sep 18 20:18:37.232646 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:37.232609    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn" event={"ID":"62918655-ac3d-4e65-b7d1-84a1b797f5a9","Type":"ContainerStarted","Data":"49ae3dc706943e390d9805fe4150fc4e354d90c8c28f52d068f3e9c672c87b7d"}
Sep 18 20:18:37.233098 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:37.232702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn" event={"ID":"62918655-ac3d-4e65-b7d1-84a1b797f5a9","Type":"ContainerStarted","Data":"9a7615539b31b9d91a23e2b6353157c7be94ee8a277ace8d39de6d35ca93c27c"}
Sep 18 20:18:37.233098 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:37.232732    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"
Sep 18 20:18:37.234438 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:37.234394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc" event={"ID":"440a5a06-5f73-40f5-b7d3-90b9f6ae925a","Type":"ContainerStarted","Data":"2b20707e3ea13f570cc34fbfa5009739f6847e4a228099be388d3556d6596f4c"}
Sep 18 20:18:37.234592 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:37.234457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc" event={"ID":"440a5a06-5f73-40f5-b7d3-90b9f6ae925a","Type":"ContainerStarted","Data":"0bdaca510af7dfa411fb33102c1ab2677d08e6cb4cc1b8c21d05ae3a0af269e7"}
Sep 18 20:18:37.234726 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:37.234699    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"
Sep 18 20:18:37.260094 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:37.260045    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn" podStartSLOduration=1.260031936 podStartE2EDuration="1.260031936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:37.25824921 +0000 UTC m=+1319.096713165" watchObservedRunningTime="2026-09-18 20:18:37.260031936 +0000 UTC m=+1319.098495914"
Sep 18 20:18:37.286711 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:37.286660    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc" podStartSLOduration=1.286642013 podStartE2EDuration="1.286642013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:37.285683577 +0000 UTC m=+1319.124147528" watchObservedRunningTime="2026-09-18 20:18:37.286642013 +0000 UTC m=+1319.125105970"
Sep 18 20:18:37.799119 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:37.799085    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:38.740810 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:38.740762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:39.637486 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:39.637450    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:40.587105 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:40.587073    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:41.488403 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:41.488369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:42.240147 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:42.240119    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:18:42.570009 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:42.569899    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:47.236565 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:47.236531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"
Sep 18 20:18:48.242033 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:48.241990    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"
Sep 18 20:18:48.242471 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:48.242196    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"
Sep 18 20:18:58.796740 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:58.796650    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:18:59.668253 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:18:59.668212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:00.552407 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:00.552370    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:00.655241 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:00.655203    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:00.967373 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:00.967339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:01.564530 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:01.564492    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:02.569051 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:02.569015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:03.478070 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:03.478033    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:04.572720 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:04.572685    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:05.258794 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:05.258763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:05.563000 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:05.562885    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:05.795444 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:05.795408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:05.952382 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:05.952294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:06.324490 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:06.324453    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:08.599402 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:08.599364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:09.318514 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:09.318478    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:09.475672 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:09.475636    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:09.868995 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:09.868952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:10.793060 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:10.793017    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:11.268794 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:11.268758    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:11.650559 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:11.650475    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:12.149367 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:12.149327    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:13.010618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:13.010575    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:13.951722 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:13.951676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:14.050864 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:14.050834    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:14.162554 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:14.162515    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:14.257145 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:14.257109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:14.370421 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:14.370388    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:14.699301 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:14.699221    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:14.803734 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:14.803702    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:15.038158 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.038127    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"]
Sep 18 20:19:15.042498 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.042477    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"
Sep 18 20:19:15.045251 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.045225    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-81f19d-dockercfg-77mhm\""
Sep 18 20:19:15.045521 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.045503    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-81f19d\""
Sep 18 20:19:15.063876 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.063844    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"]
Sep 18 20:19:15.132650 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.132622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8sfq\" (UniqueName: \"kubernetes.io/projected/d1af5b7f-f6db-4ca8-9e41-0ed3820f750d-kube-api-access-f8sfq\") pod \"payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg\" (UID: \"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"
Sep 18 20:19:15.132818 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.132749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1af5b7f-f6db-4ca8-9e41-0ed3820f750d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg\" (UID: \"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"
Sep 18 20:19:15.233850 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.233814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1af5b7f-f6db-4ca8-9e41-0ed3820f750d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg\" (UID: \"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"
Sep 18 20:19:15.233850 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.233856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8sfq\" (UniqueName: \"kubernetes.io/projected/d1af5b7f-f6db-4ca8-9e41-0ed3820f750d-kube-api-access-f8sfq\") pod \"payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg\" (UID: \"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"
Sep 18 20:19:15.234620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.234600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1af5b7f-f6db-4ca8-9e41-0ed3820f750d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg\" (UID: \"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"
Sep 18 20:19:15.247045 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.247012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8sfq\" (UniqueName: \"kubernetes.io/projected/d1af5b7f-f6db-4ca8-9e41-0ed3820f750d-kube-api-access-f8sfq\") pod \"payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg\" (UID: \"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"
Sep 18 20:19:15.352602 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.352532    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"
Sep 18 20:19:15.433211 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.433154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:15.635351 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:15.635328    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"]
Sep 18 20:19:15.637663 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:19:15.637632    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1af5b7f_f6db_4ca8_9e41_0ed3820f750d.slice/crio-6a56c59c3331275f1d39c32c3377647d9e54e715e30285b09880463311316902 WatchSource:0}: Error finding container 6a56c59c3331275f1d39c32c3377647d9e54e715e30285b09880463311316902: Status 404 returned error can't find the container with id 6a56c59c3331275f1d39c32c3377647d9e54e715e30285b09880463311316902
Sep 18 20:19:16.441584 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:16.441538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg" event={"ID":"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d","Type":"ContainerStarted","Data":"22b0b01fb96e6f11468016d66bcedeeb2f027293a2ebc6087fe3a1c92781a5e0"}
Sep 18 20:19:16.442002 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:16.441597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg" event={"ID":"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d","Type":"ContainerStarted","Data":"6a56c59c3331275f1d39c32c3377647d9e54e715e30285b09880463311316902"}
Sep 18 20:19:16.442002 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:16.441617    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"
Sep 18 20:19:16.451360 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:16.451336    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:16.469188 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:16.469141    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg" podStartSLOduration=2.469127534 podStartE2EDuration="2.469127534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:16.467014359 +0000 UTC m=+1358.305478312" watchObservedRunningTime="2026-09-18 20:19:16.469127534 +0000 UTC m=+1358.307591488"
Sep 18 20:19:17.585472 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:17.585431    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:19.069065 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:19.069027    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:20.057173 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:20.057129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:21.072285 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:21.072255    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:22.305343 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:22.305304    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:22.412219 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:22.412184    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:22.638724 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:22.638640    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:22.922941 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:22.922848    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:23.023818 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.023778    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:23.204756 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.204675    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"]
Sep 18 20:19:23.208379 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.208357    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"
Sep 18 20:19:23.211063 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.211038    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7d49a30d-dockercfg-g5fsj\""
Sep 18 20:19:23.211230 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.211078    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7d49a30d\""
Sep 18 20:19:23.221271 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.221244    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"]
Sep 18 20:19:23.312547 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.312509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjvtz\" (UniqueName: \"kubernetes.io/projected/ed9ea7ee-c52e-436a-8597-f0a599ed9afc-kube-api-access-zjvtz\") pod \"payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j\" (UID: \"ed9ea7ee-c52e-436a-8597-f0a599ed9afc\") " pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"
Sep 18 20:19:23.312920 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.312598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed9ea7ee-c52e-436a-8597-f0a599ed9afc-plugins-config-volume\") pod \"payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j\" (UID: \"ed9ea7ee-c52e-436a-8597-f0a599ed9afc\") " pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"
Sep 18 20:19:23.413375 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.413337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed9ea7ee-c52e-436a-8597-f0a599ed9afc-plugins-config-volume\") pod \"payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j\" (UID: \"ed9ea7ee-c52e-436a-8597-f0a599ed9afc\") " pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"
Sep 18 20:19:23.413536 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.413405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjvtz\" (UniqueName: \"kubernetes.io/projected/ed9ea7ee-c52e-436a-8597-f0a599ed9afc-kube-api-access-zjvtz\") pod \"payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j\" (UID: \"ed9ea7ee-c52e-436a-8597-f0a599ed9afc\") " pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"
Sep 18 20:19:23.414075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.414056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed9ea7ee-c52e-436a-8597-f0a599ed9afc-plugins-config-volume\") pod \"payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j\" (UID: \"ed9ea7ee-c52e-436a-8597-f0a599ed9afc\") " pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"
Sep 18 20:19:23.424082 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.424048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjvtz\" (UniqueName: \"kubernetes.io/projected/ed9ea7ee-c52e-436a-8597-f0a599ed9afc-kube-api-access-zjvtz\") pod \"payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j\" (UID: \"ed9ea7ee-c52e-436a-8597-f0a599ed9afc\") " pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"
Sep 18 20:19:23.458075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.458005    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:23.520590 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.520558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"
Sep 18 20:19:23.809128 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:23.806104    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"]
Sep 18 20:19:23.810883 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:19:23.810843    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9ea7ee_c52e_436a_8597_f0a599ed9afc.slice/crio-3ba9b37d0ceb9799601993cc015425783d917919d92d34e79a5c1dd50029d6aa WatchSource:0}: Error finding container 3ba9b37d0ceb9799601993cc015425783d917919d92d34e79a5c1dd50029d6aa: Status 404 returned error can't find the container with id 3ba9b37d0ceb9799601993cc015425783d917919d92d34e79a5c1dd50029d6aa
Sep 18 20:19:24.481657 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:24.481616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j" event={"ID":"ed9ea7ee-c52e-436a-8597-f0a599ed9afc","Type":"ContainerStarted","Data":"a50440fc6c91785751da74a986729f730ef6c32611305ed4a82a34ddf5eb106f"}
Sep 18 20:19:24.482057 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:24.481678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j" event={"ID":"ed9ea7ee-c52e-436a-8597-f0a599ed9afc","Type":"ContainerStarted","Data":"3ba9b37d0ceb9799601993cc015425783d917919d92d34e79a5c1dd50029d6aa"}
Sep 18 20:19:24.482057 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:24.481734    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"
Sep 18 20:19:24.502845 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:24.502804    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j" podStartSLOduration=1.502790353 podStartE2EDuration="1.502790353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:24.500246882 +0000 UTC m=+1366.338710836" watchObservedRunningTime="2026-09-18 20:19:24.502790353 +0000 UTC m=+1366.341254350"
Sep 18 20:19:24.533772 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:24.533740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:25.522717 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:25.522688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:26.429885 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:26.429844    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:27.449533 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:27.449506    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"
Sep 18 20:19:27.482395 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:27.482364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:28.573629 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:28.573593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:29.018206 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:29.018175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:29.607519 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:29.607479    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:30.554122 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:30.554081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:31.670439 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:31.670403    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:33.144660 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:33.144624    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:33.912334 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:33.912298    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:34.030363 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:34.030322    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:34.867327 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:34.867293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:35.489495 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:35.489463    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"
Sep 18 20:19:36.136977 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:36.136928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:37.369138 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:37.369105    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:38.339613 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:38.339579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:39.444743 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:39.444703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:42.143043 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:42.143010    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:45.821567 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:45.821524    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:19:52.558462 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:19:52.558423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:05.669843 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:20:05.669809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:16.540835 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:20:16.540792    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:16.639397 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:20:16.639363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:20:30.867306 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:20:30.867275    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:16.565577 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:16.565535    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:17.471766 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:17.471720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:18.276482 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:18.276443    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:19.402505 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:19.402464    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:20.288202 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:20.288167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:21.261419 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:21.261388    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:22.454898 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:22.454860    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:30.616620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:30.616581    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:31.239707 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:31.239666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:31.277788 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:31.277755    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"]
Sep 18 20:21:31.278074 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:31.278050    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j" podUID="ed9ea7ee-c52e-436a-8597-f0a599ed9afc" containerName="payload-processing" containerID="cri-o://a50440fc6c91785751da74a986729f730ef6c32611305ed4a82a34ddf5eb106f" gracePeriod=30
Sep 18 20:21:31.355120 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:31.355076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:31.513386 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:31.513305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:31.534283 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:31.534262    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"
Sep 18 20:21:31.592127 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:31.592088    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:31.663047 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:31.663019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed9ea7ee-c52e-436a-8597-f0a599ed9afc-plugins-config-volume\") pod \"ed9ea7ee-c52e-436a-8597-f0a599ed9afc\" (UID: \"ed9ea7ee-c52e-436a-8597-f0a599ed9afc\") "
Sep 18 20:21:31.663412 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:31.663096    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjvtz\" (UniqueName: \"kubernetes.io/projected/ed9ea7ee-c52e-436a-8597-f0a599ed9afc-kube-api-access-zjvtz\") pod \"ed9ea7ee-c52e-436a-8597-f0a599ed9afc\" (UID: \"ed9ea7ee-c52e-436a-8597-f0a599ed9afc\") "
Sep 18 20:21:31.663412 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:31.663370    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9ea7ee-c52e-436a-8597-f0a599ed9afc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed9ea7ee-c52e-436a-8597-f0a599ed9afc" (UID: "ed9ea7ee-c52e-436a-8597-f0a599ed9afc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:21:31.665219 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:31.665195    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9ea7ee-c52e-436a-8597-f0a599ed9afc-kube-api-access-zjvtz" (OuterVolumeSpecName: "kube-api-access-zjvtz") pod "ed9ea7ee-c52e-436a-8597-f0a599ed9afc" (UID: "ed9ea7ee-c52e-436a-8597-f0a599ed9afc"). InnerVolumeSpecName "kube-api-access-zjvtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:31.764460 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:31.764371    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjvtz\" (UniqueName: \"kubernetes.io/projected/ed9ea7ee-c52e-436a-8597-f0a599ed9afc-kube-api-access-zjvtz\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:21:31.764460 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:31.764404    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed9ea7ee-c52e-436a-8597-f0a599ed9afc-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:21:32.077514 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:32.077429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:32.124726 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:32.124696    2578 generic.go:356] "Generic (PLEG): container finished" podID="ed9ea7ee-c52e-436a-8597-f0a599ed9afc" containerID="a50440fc6c91785751da74a986729f730ef6c32611305ed4a82a34ddf5eb106f" exitCode=0
Sep 18 20:21:32.124925 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:32.124746    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"
Sep 18 20:21:32.124925 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:32.124760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j" event={"ID":"ed9ea7ee-c52e-436a-8597-f0a599ed9afc","Type":"ContainerDied","Data":"a50440fc6c91785751da74a986729f730ef6c32611305ed4a82a34ddf5eb106f"}
Sep 18 20:21:32.124925 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:32.124804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j" event={"ID":"ed9ea7ee-c52e-436a-8597-f0a599ed9afc","Type":"ContainerDied","Data":"3ba9b37d0ceb9799601993cc015425783d917919d92d34e79a5c1dd50029d6aa"}
Sep 18 20:21:32.124925 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:32.124825    2578 scope.go:117] "RemoveContainer" containerID="a50440fc6c91785751da74a986729f730ef6c32611305ed4a82a34ddf5eb106f"
Sep 18 20:21:32.134260 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:32.134239    2578 scope.go:117] "RemoveContainer" containerID="a50440fc6c91785751da74a986729f730ef6c32611305ed4a82a34ddf5eb106f"
Sep 18 20:21:32.134523 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:21:32.134505    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50440fc6c91785751da74a986729f730ef6c32611305ed4a82a34ddf5eb106f\": container with ID starting with a50440fc6c91785751da74a986729f730ef6c32611305ed4a82a34ddf5eb106f not found: ID does not exist" containerID="a50440fc6c91785751da74a986729f730ef6c32611305ed4a82a34ddf5eb106f"
Sep 18 20:21:32.134587 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:32.134535    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50440fc6c91785751da74a986729f730ef6c32611305ed4a82a34ddf5eb106f"} err="failed to get container status \"a50440fc6c91785751da74a986729f730ef6c32611305ed4a82a34ddf5eb106f\": rpc error: code = NotFound desc = could not find container \"a50440fc6c91785751da74a986729f730ef6c32611305ed4a82a34ddf5eb106f\": container with ID starting with a50440fc6c91785751da74a986729f730ef6c32611305ed4a82a34ddf5eb106f not found: ID does not exist"
Sep 18 20:21:32.154119 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:32.154096    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"]
Sep 18 20:21:32.163700 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:32.163680    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d49a30d-db75dd9fc-6z26j"]
Sep 18 20:21:32.320144 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:32.320111    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed9ea7ee-c52e-436a-8597-f0a599ed9afc" path="/var/lib/kubelet/pods/ed9ea7ee-c52e-436a-8597-f0a599ed9afc/volumes"
Sep 18 20:21:32.963554 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:32.963517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:33.749370 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:33.749332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:34.672877 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:34.672826    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:34.773262 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:34.773229    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:35.555893 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:35.555855    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:36.537159 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:36.537122    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:37.335213 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:37.335175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:39.838817 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:39.838750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:21:59.110481 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:21:59.110436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:00.277074 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:00.277029    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:01.108453 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:01.108411    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:02.132721 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:02.132688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:03.052094 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:03.052054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:04.055809 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:04.055772    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"]
Sep 18 20:22:04.075641 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:04.075610    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"]
Sep 18 20:22:04.150782 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:04.150742    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:04.251101 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:04.251069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:05.053219 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:05.053177    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:09.650191 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:09.650155    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:09.729741 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:09.729706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:14.836029 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:14.835987    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"]
Sep 18 20:22:14.836503 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:14.836241    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2" podUID="9f03d8a5-81b3-42d9-b68f-0e7771c04658" containerName="maas-api" containerID="cri-o://9c8004e4c31ad4959386ea6560f319794f6d797aa9e36d44a5f7c87453953071" gracePeriod=30
Sep 18 20:22:14.872093 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:14.872058    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5284dc-29829375-fmf6t"]
Sep 18 20:22:14.873183 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:14.873158    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:14.875053 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:14.875029    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5284dc-29829375-fmf6t"]
Sep 18 20:22:14.897594 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:14.897562    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"]
Sep 18 20:22:14.897786 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:14.897759    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756" podUID="970fbd29-7b59-452e-99e2-4d351fca4188" containerName="payload-processing" containerID="cri-o://01f8883cee42036c9b486b9614f50e144250580dafc3a1dc8e4ad64b5974ccc1" gracePeriod=30
Sep 18 20:22:14.965164 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:14.965120    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2" podUID="9f03d8a5-81b3-42d9-b68f-0e7771c04658" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.34:8443/health\": dial tcp 10.132.0.34:8443: connect: connection refused"
Sep 18 20:22:14.974245 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:14.974199    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:15.075172 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.075135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:15.152195 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.152170    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:22:15.179180 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.179065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:15.182803 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.182769    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"
Sep 18 20:22:15.271742 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.271702    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tsc2\" (UniqueName: \"kubernetes.io/projected/970fbd29-7b59-452e-99e2-4d351fca4188-kube-api-access-2tsc2\") pod \"970fbd29-7b59-452e-99e2-4d351fca4188\" (UID: \"970fbd29-7b59-452e-99e2-4d351fca4188\") "
Sep 18 20:22:15.271742 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.271755    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f03d8a5-81b3-42d9-b68f-0e7771c04658-maas-api-tls\") pod \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") "
Sep 18 20:22:15.272017 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.271776    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47jjj\" (UniqueName: \"kubernetes.io/projected/9f03d8a5-81b3-42d9-b68f-0e7771c04658-kube-api-access-47jjj\") pod \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") "
Sep 18 20:22:15.272017 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.271798    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970fbd29-7b59-452e-99e2-4d351fca4188-plugins-config-volume\") pod \"970fbd29-7b59-452e-99e2-4d351fca4188\" (UID: \"970fbd29-7b59-452e-99e2-4d351fca4188\") "
Sep 18 20:22:15.272017 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.271845    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f03d8a5-81b3-42d9-b68f-0e7771c04658-openshift-service-ca\") pod \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") "
Sep 18 20:22:15.272017 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.271920    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f03d8a5-81b3-42d9-b68f-0e7771c04658-metrics-certs\") pod \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\" (UID: \"9f03d8a5-81b3-42d9-b68f-0e7771c04658\") "
Sep 18 20:22:15.272296 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.272269    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970fbd29-7b59-452e-99e2-4d351fca4188-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "970fbd29-7b59-452e-99e2-4d351fca4188" (UID: "970fbd29-7b59-452e-99e2-4d351fca4188"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:15.272361 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.272281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f03d8a5-81b3-42d9-b68f-0e7771c04658-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f03d8a5-81b3-42d9-b68f-0e7771c04658" (UID: "9f03d8a5-81b3-42d9-b68f-0e7771c04658"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:15.274126 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.274098    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f03d8a5-81b3-42d9-b68f-0e7771c04658-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f03d8a5-81b3-42d9-b68f-0e7771c04658" (UID: "9f03d8a5-81b3-42d9-b68f-0e7771c04658"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:15.274236 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.274102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970fbd29-7b59-452e-99e2-4d351fca4188-kube-api-access-2tsc2" (OuterVolumeSpecName: "kube-api-access-2tsc2") pod "970fbd29-7b59-452e-99e2-4d351fca4188" (UID: "970fbd29-7b59-452e-99e2-4d351fca4188"). InnerVolumeSpecName "kube-api-access-2tsc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:15.274236 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.274166    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f03d8a5-81b3-42d9-b68f-0e7771c04658-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9f03d8a5-81b3-42d9-b68f-0e7771c04658" (UID: "9f03d8a5-81b3-42d9-b68f-0e7771c04658"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:15.274236 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.274176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f03d8a5-81b3-42d9-b68f-0e7771c04658-kube-api-access-47jjj" (OuterVolumeSpecName: "kube-api-access-47jjj") pod "9f03d8a5-81b3-42d9-b68f-0e7771c04658" (UID: "9f03d8a5-81b3-42d9-b68f-0e7771c04658"). InnerVolumeSpecName "kube-api-access-47jjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:15.341813 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.341779    2578 generic.go:356] "Generic (PLEG): container finished" podID="970fbd29-7b59-452e-99e2-4d351fca4188" containerID="01f8883cee42036c9b486b9614f50e144250580dafc3a1dc8e4ad64b5974ccc1" exitCode=0
Sep 18 20:22:15.341983 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.341832    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"
Sep 18 20:22:15.341983 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.341867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756" event={"ID":"970fbd29-7b59-452e-99e2-4d351fca4188","Type":"ContainerDied","Data":"01f8883cee42036c9b486b9614f50e144250580dafc3a1dc8e4ad64b5974ccc1"}
Sep 18 20:22:15.341983 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.341926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756" event={"ID":"970fbd29-7b59-452e-99e2-4d351fca4188","Type":"ContainerDied","Data":"1ff2b996a191e30efb9cc8c12c0e1babb0f79335c339c03cbe08d4649065dc4c"}
Sep 18 20:22:15.342139 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.341990    2578 scope.go:117] "RemoveContainer" containerID="01f8883cee42036c9b486b9614f50e144250580dafc3a1dc8e4ad64b5974ccc1"
Sep 18 20:22:15.343599 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.343574    2578 generic.go:356] "Generic (PLEG): container finished" podID="9f03d8a5-81b3-42d9-b68f-0e7771c04658" containerID="9c8004e4c31ad4959386ea6560f319794f6d797aa9e36d44a5f7c87453953071" exitCode=0
Sep 18 20:22:15.343692 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.343618    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"
Sep 18 20:22:15.343692 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.343607    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2" event={"ID":"9f03d8a5-81b3-42d9-b68f-0e7771c04658","Type":"ContainerDied","Data":"9c8004e4c31ad4959386ea6560f319794f6d797aa9e36d44a5f7c87453953071"}
Sep 18 20:22:15.343778 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.343707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2" event={"ID":"9f03d8a5-81b3-42d9-b68f-0e7771c04658","Type":"ContainerDied","Data":"7765318cf1539f70950848e94f128da8f9912c617f7306fb95572d25430960e2"}
Sep 18 20:22:15.351243 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.351126    2578 scope.go:117] "RemoveContainer" containerID="01f8883cee42036c9b486b9614f50e144250580dafc3a1dc8e4ad64b5974ccc1"
Sep 18 20:22:15.351480 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:22:15.351458    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f8883cee42036c9b486b9614f50e144250580dafc3a1dc8e4ad64b5974ccc1\": container with ID starting with 01f8883cee42036c9b486b9614f50e144250580dafc3a1dc8e4ad64b5974ccc1 not found: ID does not exist" containerID="01f8883cee42036c9b486b9614f50e144250580dafc3a1dc8e4ad64b5974ccc1"
Sep 18 20:22:15.351553 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.351487    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f8883cee42036c9b486b9614f50e144250580dafc3a1dc8e4ad64b5974ccc1"} err="failed to get container status \"01f8883cee42036c9b486b9614f50e144250580dafc3a1dc8e4ad64b5974ccc1\": rpc error: code = NotFound desc = could not find container \"01f8883cee42036c9b486b9614f50e144250580dafc3a1dc8e4ad64b5974ccc1\": container with ID starting with 01f8883cee42036c9b486b9614f50e144250580dafc3a1dc8e4ad64b5974ccc1 not found: ID does not exist"
Sep 18 20:22:15.351553 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.351504    2578 scope.go:117] "RemoveContainer" containerID="9c8004e4c31ad4959386ea6560f319794f6d797aa9e36d44a5f7c87453953071"
Sep 18 20:22:15.359630 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.359613    2578 scope.go:117] "RemoveContainer" containerID="9c8004e4c31ad4959386ea6560f319794f6d797aa9e36d44a5f7c87453953071"
Sep 18 20:22:15.359894 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:22:15.359873    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8004e4c31ad4959386ea6560f319794f6d797aa9e36d44a5f7c87453953071\": container with ID starting with 9c8004e4c31ad4959386ea6560f319794f6d797aa9e36d44a5f7c87453953071 not found: ID does not exist" containerID="9c8004e4c31ad4959386ea6560f319794f6d797aa9e36d44a5f7c87453953071"
Sep 18 20:22:15.359982 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.359903    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8004e4c31ad4959386ea6560f319794f6d797aa9e36d44a5f7c87453953071"} err="failed to get container status \"9c8004e4c31ad4959386ea6560f319794f6d797aa9e36d44a5f7c87453953071\": rpc error: code = NotFound desc = could not find container \"9c8004e4c31ad4959386ea6560f319794f6d797aa9e36d44a5f7c87453953071\": container with ID starting with 9c8004e4c31ad4959386ea6560f319794f6d797aa9e36d44a5f7c87453953071 not found: ID does not exist"
Sep 18 20:22:15.369143 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.369120    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"]
Sep 18 20:22:15.373375 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.373356    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f03d8a5-81b3-42d9-b68f-0e7771c04658-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.373462 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.373377    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tsc2\" (UniqueName: \"kubernetes.io/projected/970fbd29-7b59-452e-99e2-4d351fca4188-kube-api-access-2tsc2\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.373462 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.373392    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f03d8a5-81b3-42d9-b68f-0e7771c04658-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.373462 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.373405    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47jjj\" (UniqueName: \"kubernetes.io/projected/9f03d8a5-81b3-42d9-b68f-0e7771c04658-kube-api-access-47jjj\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.373462 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.373417    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970fbd29-7b59-452e-99e2-4d351fca4188-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.373462 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.373430    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f03d8a5-81b3-42d9-b68f-0e7771c04658-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:15.376353 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.376330    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5284dc-5754465775-hc8p2"]
Sep 18 20:22:15.391467 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.391446    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"]
Sep 18 20:22:15.397958 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.397928    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5284dc-97575bc5-jl756"]
Sep 18 20:22:15.509760 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.509727    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:15.689161 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:15.689077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:16.319671 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:16.319633    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88063197-3665-4a07-94e5-06aa35d058f7" path="/var/lib/kubelet/pods/88063197-3665-4a07-94e5-06aa35d058f7/volumes"
Sep 18 20:22:16.320045 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:16.319985    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="970fbd29-7b59-452e-99e2-4d351fca4188" path="/var/lib/kubelet/pods/970fbd29-7b59-452e-99e2-4d351fca4188/volumes"
Sep 18 20:22:16.320354 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:16.320341    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f03d8a5-81b3-42d9-b68f-0e7771c04658" path="/var/lib/kubelet/pods/9f03d8a5-81b3-42d9-b68f-0e7771c04658/volumes"
Sep 18 20:22:16.461086 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:16.461049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:16.756287 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:16.756250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:17.757655 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:17.757623    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:18.321789 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:18.321746    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:18.654448 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:18.654356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:19.075573 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.075533    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8" podUID="77f8cb7f-4b15-4e77-a35e-18e87901082a" containerName="main" containerID="cri-o://7d0e4119414685269eac3ca32ad817abd735f2894ca244eda0398fbc24043ebd" gracePeriod=45
Sep 18 20:22:19.096377 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.096325    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg" podUID="4f109e24-48d7-4407-8b0a-18c022d517b4" containerName="main" containerID="cri-o://7bd3743e6d4138e33509bd186b651c7d6967e21cc50524f8f295a12dfac3d558" gracePeriod=45
Sep 18 20:22:19.370284 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.370254    2578 generic.go:356] "Generic (PLEG): container finished" podID="4f109e24-48d7-4407-8b0a-18c022d517b4" containerID="7bd3743e6d4138e33509bd186b651c7d6967e21cc50524f8f295a12dfac3d558" exitCode=0
Sep 18 20:22:19.370413 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.370319    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg" event={"ID":"4f109e24-48d7-4407-8b0a-18c022d517b4","Type":"ContainerDied","Data":"7bd3743e6d4138e33509bd186b651c7d6967e21cc50524f8f295a12dfac3d558"}
Sep 18 20:22:19.370413 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.370366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg" event={"ID":"4f109e24-48d7-4407-8b0a-18c022d517b4","Type":"ContainerDied","Data":"58a703f4802952881f6f44a5a51c458b885cfe7aed62ce370ad0db25b3153a46"}
Sep 18 20:22:19.370413 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.370382    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58a703f4802952881f6f44a5a51c458b885cfe7aed62ce370ad0db25b3153a46"
Sep 18 20:22:19.371720 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.371697    2578 generic.go:356] "Generic (PLEG): container finished" podID="77f8cb7f-4b15-4e77-a35e-18e87901082a" containerID="7d0e4119414685269eac3ca32ad817abd735f2894ca244eda0398fbc24043ebd" exitCode=0
Sep 18 20:22:19.371806 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.371739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8" event={"ID":"77f8cb7f-4b15-4e77-a35e-18e87901082a","Type":"ContainerDied","Data":"7d0e4119414685269eac3ca32ad817abd735f2894ca244eda0398fbc24043ebd"}
Sep 18 20:22:19.371806 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.371762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8" event={"ID":"77f8cb7f-4b15-4e77-a35e-18e87901082a","Type":"ContainerDied","Data":"69e39548df54990cca69ac297e5434b66773af34f74478a2fc544e1d9b13d078"}
Sep 18 20:22:19.371806 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.371774    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69e39548df54990cca69ac297e5434b66773af34f74478a2fc544e1d9b13d078"
Sep 18 20:22:19.373386 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.373366    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:22:19.376637 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.376618    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:22:19.508844 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.508812    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk7f8\" (UniqueName: \"kubernetes.io/projected/4f109e24-48d7-4407-8b0a-18c022d517b4-kube-api-access-lk7f8\") pod \"4f109e24-48d7-4407-8b0a-18c022d517b4\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") "
Sep 18 20:22:19.508844 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.508849    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-model-cache\") pod \"77f8cb7f-4b15-4e77-a35e-18e87901082a\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") "
Sep 18 20:22:19.509115 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.508876    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-dshm\") pod \"77f8cb7f-4b15-4e77-a35e-18e87901082a\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") "
Sep 18 20:22:19.509115 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.508897    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-dshm\") pod \"4f109e24-48d7-4407-8b0a-18c022d517b4\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") "
Sep 18 20:22:19.509115 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.508953    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9djx\" (UniqueName: \"kubernetes.io/projected/77f8cb7f-4b15-4e77-a35e-18e87901082a-kube-api-access-w9djx\") pod \"77f8cb7f-4b15-4e77-a35e-18e87901082a\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") "
Sep 18 20:22:19.509115 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.508977    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f109e24-48d7-4407-8b0a-18c022d517b4-tls-certs\") pod \"4f109e24-48d7-4407-8b0a-18c022d517b4\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") "
Sep 18 20:22:19.509115 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.508995    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-model-cache\") pod \"4f109e24-48d7-4407-8b0a-18c022d517b4\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") "
Sep 18 20:22:19.509390 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509126    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-model-cache" (OuterVolumeSpecName: "model-cache") pod "77f8cb7f-4b15-4e77-a35e-18e87901082a" (UID: "77f8cb7f-4b15-4e77-a35e-18e87901082a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:19.509390 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509137    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-home\") pod \"4f109e24-48d7-4407-8b0a-18c022d517b4\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") "
Sep 18 20:22:19.509390 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509194    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-tmp-dir\") pod \"77f8cb7f-4b15-4e77-a35e-18e87901082a\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") "
Sep 18 20:22:19.509390 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509244    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-tmp-dir\") pod \"4f109e24-48d7-4407-8b0a-18c022d517b4\" (UID: \"4f109e24-48d7-4407-8b0a-18c022d517b4\") "
Sep 18 20:22:19.509390 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509260    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f109e24-48d7-4407-8b0a-18c022d517b4" (UID: "4f109e24-48d7-4407-8b0a-18c022d517b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:19.509390 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509269    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-home\") pod \"77f8cb7f-4b15-4e77-a35e-18e87901082a\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") "
Sep 18 20:22:19.509390 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509377    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f8cb7f-4b15-4e77-a35e-18e87901082a-tls-certs\") pod \"77f8cb7f-4b15-4e77-a35e-18e87901082a\" (UID: \"77f8cb7f-4b15-4e77-a35e-18e87901082a\") "
Sep 18 20:22:19.509754 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509451    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-home" (OuterVolumeSpecName: "home") pod "77f8cb7f-4b15-4e77-a35e-18e87901082a" (UID: "77f8cb7f-4b15-4e77-a35e-18e87901082a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:19.509754 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77f8cb7f-4b15-4e77-a35e-18e87901082a" (UID: "77f8cb7f-4b15-4e77-a35e-18e87901082a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:19.509754 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f109e24-48d7-4407-8b0a-18c022d517b4" (UID: "4f109e24-48d7-4407-8b0a-18c022d517b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:19.509754 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509746    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-model-cache\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.510001 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509763    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-model-cache\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.510001 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509775    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-tmp-dir\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.510001 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509787    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-tmp-dir\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.510001 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509799    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-home\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.510001 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.509799    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-home" (OuterVolumeSpecName: "home") pod "4f109e24-48d7-4407-8b0a-18c022d517b4" (UID: "4f109e24-48d7-4407-8b0a-18c022d517b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:19.511953 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.511751    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-dshm" (OuterVolumeSpecName: "dshm") pod "77f8cb7f-4b15-4e77-a35e-18e87901082a" (UID: "77f8cb7f-4b15-4e77-a35e-18e87901082a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:19.511953 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.511760    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f109e24-48d7-4407-8b0a-18c022d517b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f109e24-48d7-4407-8b0a-18c022d517b4" (UID: "4f109e24-48d7-4407-8b0a-18c022d517b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:19.511953 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.511838    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f109e24-48d7-4407-8b0a-18c022d517b4-kube-api-access-lk7f8" (OuterVolumeSpecName: "kube-api-access-lk7f8") pod "4f109e24-48d7-4407-8b0a-18c022d517b4" (UID: "4f109e24-48d7-4407-8b0a-18c022d517b4"). InnerVolumeSpecName "kube-api-access-lk7f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:19.511953 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.511886    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f8cb7f-4b15-4e77-a35e-18e87901082a-kube-api-access-w9djx" (OuterVolumeSpecName: "kube-api-access-w9djx") pod "77f8cb7f-4b15-4e77-a35e-18e87901082a" (UID: "77f8cb7f-4b15-4e77-a35e-18e87901082a"). InnerVolumeSpecName "kube-api-access-w9djx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:19.512468 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.512053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f8cb7f-4b15-4e77-a35e-18e87901082a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77f8cb7f-4b15-4e77-a35e-18e87901082a" (UID: "77f8cb7f-4b15-4e77-a35e-18e87901082a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:19.512468 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.512056    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-dshm" (OuterVolumeSpecName: "dshm") pod "4f109e24-48d7-4407-8b0a-18c022d517b4" (UID: "4f109e24-48d7-4407-8b0a-18c022d517b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:19.571204 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.571165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:19.610565 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.610474    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f8cb7f-4b15-4e77-a35e-18e87901082a-tls-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.610565 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.610510    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk7f8\" (UniqueName: \"kubernetes.io/projected/4f109e24-48d7-4407-8b0a-18c022d517b4-kube-api-access-lk7f8\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.610565 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.610520    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f8cb7f-4b15-4e77-a35e-18e87901082a-dshm\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.610565 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.610529    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-dshm\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.610565 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.610537    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9djx\" (UniqueName: \"kubernetes.io/projected/77f8cb7f-4b15-4e77-a35e-18e87901082a-kube-api-access-w9djx\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.610565 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.610547    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f109e24-48d7-4407-8b0a-18c022d517b4-tls-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:19.610565 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:19.610555    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f109e24-48d7-4407-8b0a-18c022d517b4-home\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:22:20.376130 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:20.376100    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"
Sep 18 20:22:20.376538 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:20.376100    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"
Sep 18 20:22:20.395554 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:20.395529    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"]
Sep 18 20:22:20.400026 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:20.399999    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/premium-w0-5284dc-kserve-565f857d8b-k9csg"]
Sep 18 20:22:20.410135 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:20.410112    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"]
Sep 18 20:22:20.412279 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:20.412259    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5284dc/distinct-w0-5284dc-kserve-7b4b5fd569-6vcs8"]
Sep 18 20:22:22.034846 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:22.034813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:22.165426 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:22.165383    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:22.320990 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:22.320911    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f109e24-48d7-4407-8b0a-18c022d517b4" path="/var/lib/kubelet/pods/4f109e24-48d7-4407-8b0a-18c022d517b4/volumes"
Sep 18 20:22:22.321345 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:22.321331    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77f8cb7f-4b15-4e77-a35e-18e87901082a" path="/var/lib/kubelet/pods/77f8cb7f-4b15-4e77-a35e-18e87901082a/volumes"
Sep 18 20:22:22.353366 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:22.353340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:24.637058 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:24.637022    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:24.741032 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:24.740996    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:24.929776 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:24.929696    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:26.173779 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:26.173745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:26.275060 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:26.275024    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:27.005378 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:27.005346    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:27.105462 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:27.105421    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:28.630645 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:28.630615    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:33.021285 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:33.021249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:34.751624 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:34.751589    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:34.851824 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:34.851795    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:35.458020 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:35.457979    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:40.427885 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:40.427851    2578 scope.go:117] "RemoveContainer" containerID="ec033af44f3a02bdd6ce0709d9e594a31e29c3163d8ba714e50cd7ad23eb1e6e"
Sep 18 20:22:40.436404 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:40.436380    2578 scope.go:117] "RemoveContainer" containerID="7bd3743e6d4138e33509bd186b651c7d6967e21cc50524f8f295a12dfac3d558"
Sep 18 20:22:41.773917 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:41.773884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:41.872720 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:41.872688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:42.540317 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:42.540282    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:43.004308 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:43.004279    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:43.078186 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:43.078153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:43.203617 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:43.203582    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:43.309270 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:43.309172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:53.742514 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:53.742474    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:53.971718 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:53.971678    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:22:57.501700 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:22:57.501657    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:01.555091 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:01.555050    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:03.388022 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:03.387989    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:14.883564 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:14.883527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:20.759488 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:20.759456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:21.990319 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:21.990283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:25.262982 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:25.262920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:25.463051 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:25.463011    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:26.394819 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:26.394786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:27.273663 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:27.273631    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:28.140860 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:28.140827    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:29.101571 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.101537    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"]
Sep 18 20:23:29.102591 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102562    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f03d8a5-81b3-42d9-b68f-0e7771c04658" containerName="maas-api"
Sep 18 20:23:29.102591 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102592    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f03d8a5-81b3-42d9-b68f-0e7771c04658" containerName="maas-api"
Sep 18 20:23:29.102820 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102606    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed9ea7ee-c52e-436a-8597-f0a599ed9afc" containerName="payload-processing"
Sep 18 20:23:29.102820 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102616    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed9ea7ee-c52e-436a-8597-f0a599ed9afc" containerName="payload-processing"
Sep 18 20:23:29.102820 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102670    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77f8cb7f-4b15-4e77-a35e-18e87901082a" containerName="main"
Sep 18 20:23:29.102820 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102680    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="77f8cb7f-4b15-4e77-a35e-18e87901082a" containerName="main"
Sep 18 20:23:29.102820 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102703    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f109e24-48d7-4407-8b0a-18c022d517b4" containerName="main"
Sep 18 20:23:29.102820 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102712    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f109e24-48d7-4407-8b0a-18c022d517b4" containerName="main"
Sep 18 20:23:29.102820 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102730    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="970fbd29-7b59-452e-99e2-4d351fca4188" containerName="payload-processing"
Sep 18 20:23:29.102820 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102738    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="970fbd29-7b59-452e-99e2-4d351fca4188" containerName="payload-processing"
Sep 18 20:23:29.103260 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102902    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f03d8a5-81b3-42d9-b68f-0e7771c04658" containerName="maas-api"
Sep 18 20:23:29.103260 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102915    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="77f8cb7f-4b15-4e77-a35e-18e87901082a" containerName="main"
Sep 18 20:23:29.103260 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102949    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="970fbd29-7b59-452e-99e2-4d351fca4188" containerName="payload-processing"
Sep 18 20:23:29.103260 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102961    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f109e24-48d7-4407-8b0a-18c022d517b4" containerName="main"
Sep 18 20:23:29.103260 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.102969    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed9ea7ee-c52e-436a-8597-f0a599ed9afc" containerName="payload-processing"
Sep 18 20:23:29.110266 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.110244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.114868 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.114846    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-69f1ba-serving-cert\""
Sep 18 20:23:29.121217 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.121195    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"]
Sep 18 20:23:29.170011 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.169979    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:29.215594 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.215559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gncrt\" (UniqueName: \"kubernetes.io/projected/21a2904f-9895-4d39-abc2-96e69deec110-kube-api-access-gncrt\") pod \"maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.215763 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.215611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-maas-api-tls\") pod \"maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.215763 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.215631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2904f-9895-4d39-abc2-96e69deec110-openshift-service-ca\") pod \"maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.215763 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.215721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-metrics-certs\") pod \"maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.266400 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.266363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:29.316606 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.316573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncrt\" (UniqueName: \"kubernetes.io/projected/21a2904f-9895-4d39-abc2-96e69deec110-kube-api-access-gncrt\") pod \"maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.316766 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.316620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-maas-api-tls\") pod \"maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.316766 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.316639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2904f-9895-4d39-abc2-96e69deec110-openshift-service-ca\") pod \"maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.316766 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.316679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-metrics-certs\") pod \"maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.316766 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:23:29.316746    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-serving-cert: secret "maas-api-e2e-api-a-69f1ba-serving-cert" not found
Sep 18 20:23:29.317015 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:23:29.316838    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-maas-api-tls podName:21a2904f-9895-4d39-abc2-96e69deec110 nodeName:}" failed. No retries permitted until 2026-09-18 20:23:29.816814809 +0000 UTC m=+1611.655278741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-maas-api-tls") pod "maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp" (UID: "21a2904f-9895-4d39-abc2-96e69deec110") : secret "maas-api-e2e-api-a-69f1ba-serving-cert" not found
Sep 18 20:23:29.317452 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.317434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2904f-9895-4d39-abc2-96e69deec110-openshift-service-ca\") pod \"maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.319265 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.319244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-metrics-certs\") pod \"maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.325727 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.325704    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncrt\" (UniqueName: \"kubernetes.io/projected/21a2904f-9895-4d39-abc2-96e69deec110-kube-api-access-gncrt\") pod \"maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.493238 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.493199    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:29.753329 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.753236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:29.820669 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.820618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-maas-api-tls\") pod \"maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.823176 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.823154    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-maas-api-tls\") pod \"maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:29.839183 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.839153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:29.978160 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.978121    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"]
Sep 18 20:23:29.981912 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.981891    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"
Sep 18 20:23:29.986507 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.986487    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-69f1ba\""
Sep 18 20:23:29.991920 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:29.991900    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-69f1ba-dockercfg-vg8rs\""
Sep 18 20:23:30.021380 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.021300    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:30.022730 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.022698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st28q\" (UniqueName: \"kubernetes.io/projected/2220fd64-8942-4de9-98ce-448473f3406a-kube-api-access-st28q\") pod \"payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd\" (UID: \"2220fd64-8942-4de9-98ce-448473f3406a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"
Sep 18 20:23:30.022882 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.022751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2220fd64-8942-4de9-98ce-448473f3406a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd\" (UID: \"2220fd64-8942-4de9-98ce-448473f3406a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"
Sep 18 20:23:30.033887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.033860    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"]
Sep 18 20:23:30.076454 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.076416    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"]
Sep 18 20:23:30.081022 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.080993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"
Sep 18 20:23:30.099653 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.099578    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"]
Sep 18 20:23:30.124122 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.124086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2220fd64-8942-4de9-98ce-448473f3406a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd\" (UID: \"2220fd64-8942-4de9-98ce-448473f3406a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"
Sep 18 20:23:30.124273 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.124132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk2m9\" (UniqueName: \"kubernetes.io/projected/cb8e9951-7309-482a-8e71-c7504575f7f5-kube-api-access-lk2m9\") pod \"payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz\" (UID: \"cb8e9951-7309-482a-8e71-c7504575f7f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"
Sep 18 20:23:30.124273 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.124230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8e9951-7309-482a-8e71-c7504575f7f5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz\" (UID: \"cb8e9951-7309-482a-8e71-c7504575f7f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"
Sep 18 20:23:30.124391 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.124316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st28q\" (UniqueName: \"kubernetes.io/projected/2220fd64-8942-4de9-98ce-448473f3406a-kube-api-access-st28q\") pod \"payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd\" (UID: \"2220fd64-8942-4de9-98ce-448473f3406a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"
Sep 18 20:23:30.124774 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.124753    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2220fd64-8942-4de9-98ce-448473f3406a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd\" (UID: \"2220fd64-8942-4de9-98ce-448473f3406a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"
Sep 18 20:23:30.146685 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.146662    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st28q\" (UniqueName: \"kubernetes.io/projected/2220fd64-8942-4de9-98ce-448473f3406a-kube-api-access-st28q\") pod \"payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd\" (UID: \"2220fd64-8942-4de9-98ce-448473f3406a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"
Sep 18 20:23:30.225305 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.225272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8e9951-7309-482a-8e71-c7504575f7f5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz\" (UID: \"cb8e9951-7309-482a-8e71-c7504575f7f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"
Sep 18 20:23:30.225724 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.225341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk2m9\" (UniqueName: \"kubernetes.io/projected/cb8e9951-7309-482a-8e71-c7504575f7f5-kube-api-access-lk2m9\") pod \"payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz\" (UID: \"cb8e9951-7309-482a-8e71-c7504575f7f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"
Sep 18 20:23:30.225959 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.225916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8e9951-7309-482a-8e71-c7504575f7f5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz\" (UID: \"cb8e9951-7309-482a-8e71-c7504575f7f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"
Sep 18 20:23:30.236343 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.236322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk2m9\" (UniqueName: \"kubernetes.io/projected/cb8e9951-7309-482a-8e71-c7504575f7f5-kube-api-access-lk2m9\") pod \"payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz\" (UID: \"cb8e9951-7309-482a-8e71-c7504575f7f5\") " pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"
Sep 18 20:23:30.292724 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.292656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"
Sep 18 20:23:30.311815 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.311792    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"]
Sep 18 20:23:30.314097 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:23:30.314068    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a2904f_9895_4d39_abc2_96e69deec110.slice/crio-7dbec74db9fdc69ecd09f030c46603d2695fa437c656f039778419f00d90fdd3 WatchSource:0}: Error finding container 7dbec74db9fdc69ecd09f030c46603d2695fa437c656f039778419f00d90fdd3: Status 404 returned error can't find the container with id 7dbec74db9fdc69ecd09f030c46603d2695fa437c656f039778419f00d90fdd3
Sep 18 20:23:30.315557 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.315531    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:23:30.395301 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.395271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"
Sep 18 20:23:30.573604 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.573578    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"]
Sep 18 20:23:30.575751 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:23:30.575714    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2220fd64_8942_4de9_98ce_448473f3406a.slice/crio-6f76aa9ecb00b828e60e9440d0efef41e8a08658cf43668643baece51c9b43b1 WatchSource:0}: Error finding container 6f76aa9ecb00b828e60e9440d0efef41e8a08658cf43668643baece51c9b43b1: Status 404 returned error can't find the container with id 6f76aa9ecb00b828e60e9440d0efef41e8a08658cf43668643baece51c9b43b1
Sep 18 20:23:30.678650 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.678559    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"]
Sep 18 20:23:30.682760 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:23:30.682734    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8e9951_7309_482a_8e71_c7504575f7f5.slice/crio-5dc1b07b5cbcc78c1881a301703e466b4b47a0a462834b201b59b4d7c0b14933 WatchSource:0}: Error finding container 5dc1b07b5cbcc78c1881a301703e466b4b47a0a462834b201b59b4d7c0b14933: Status 404 returned error can't find the container with id 5dc1b07b5cbcc78c1881a301703e466b4b47a0a462834b201b59b4d7c0b14933
Sep 18 20:23:30.724455 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.724422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp" event={"ID":"21a2904f-9895-4d39-abc2-96e69deec110","Type":"ContainerStarted","Data":"7dbec74db9fdc69ecd09f030c46603d2695fa437c656f039778419f00d90fdd3"}
Sep 18 20:23:30.726095 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.726070    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd" event={"ID":"2220fd64-8942-4de9-98ce-448473f3406a","Type":"ContainerStarted","Data":"8525a4af892b723db8f93049ccddd27817c64a7cf023a17d3f440e47e671ebbf"}
Sep 18 20:23:30.726219 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.726128    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"
Sep 18 20:23:30.726219 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.726152    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd" event={"ID":"2220fd64-8942-4de9-98ce-448473f3406a","Type":"ContainerStarted","Data":"6f76aa9ecb00b828e60e9440d0efef41e8a08658cf43668643baece51c9b43b1"}
Sep 18 20:23:30.727250 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.727225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz" event={"ID":"cb8e9951-7309-482a-8e71-c7504575f7f5","Type":"ContainerStarted","Data":"5dc1b07b5cbcc78c1881a301703e466b4b47a0a462834b201b59b4d7c0b14933"}
Sep 18 20:23:30.749108 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:30.749068    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd" podStartSLOduration=1.749053683 podStartE2EDuration="1.749053683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:30.748765578 +0000 UTC m=+1612.587229531" watchObservedRunningTime="2026-09-18 20:23:30.749053683 +0000 UTC m=+1612.587517640"
Sep 18 20:23:31.734894 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:31.734855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp" event={"ID":"21a2904f-9895-4d39-abc2-96e69deec110","Type":"ContainerStarted","Data":"af6816bc3e329631c691f09a8f1c52da9052fd7953e4494fdf86a8a5e412721d"}
Sep 18 20:23:31.735492 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:31.734924    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:31.736619 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:31.736595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz" event={"ID":"cb8e9951-7309-482a-8e71-c7504575f7f5","Type":"ContainerStarted","Data":"14956dc2d63982c087ff1297e412e92b0cd2896275f07c56515f917608a30e9c"}
Sep 18 20:23:31.736713 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:31.736642    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"
Sep 18 20:23:31.756448 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:31.756406    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp" podStartSLOduration=2.334271994 podStartE2EDuration="2.756392561s" totalImagesPullingTime="422.120567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:31.755232051 +0000 UTC m=+1613.593696004" watchObservedRunningTime="2026-09-18 20:23:31.756392561 +0000 UTC m=+1613.594856513"
Sep 18 20:23:31.772543 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:31.772493    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz" podStartSLOduration=1.7724757150000001 podStartE2EDuration="1.772475715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:31.771994075 +0000 UTC m=+1613.610458027" watchObservedRunningTime="2026-09-18 20:23:31.772475715 +0000 UTC m=+1613.610939669"
Sep 18 20:23:37.747329 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:37.747294    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:23:40.470435 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:40.470403    2578 scope.go:117] "RemoveContainer" containerID="7d0e4119414685269eac3ca32ad817abd735f2894ca244eda0398fbc24043ebd"
Sep 18 20:23:40.534526 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:40.534487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:41.739255 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:41.739223    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"
Sep 18 20:23:42.743750 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:42.743718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"
Sep 18 20:23:43.553626 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:43.553588    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:51.147859 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:51.147821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:52.575861 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:52.575826    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:53.309810 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:53.309774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:54.385193 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:54.385163    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:55.039987 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:55.039927    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:55.317690 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:55.317606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:55.521186 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:55.521154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:55.859120 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:55.859088    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:55.959509 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:55.959472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:56.354387 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:56.354357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:57.125585 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:57.125549    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:58.554268 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:58.554232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:59.218416 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:59.218365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:59.309375 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:59.309338    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:23:59.508281 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:23:59.508191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:00.549170 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:00.549129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:01.435189 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:01.435156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:02.202397 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:02.202353    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:02.974851 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:02.974809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:03.847159 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:03.847126    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:03.912988 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:03.912916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:04.013854 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.013818    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:04.403112 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.403072    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:04.501478 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.501447    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:04.606630 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.606597    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"]
Sep 18 20:24:04.609927 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.609911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"
Sep 18 20:24:04.612287 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.612261    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4cd712\""
Sep 18 20:24:04.612287 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.612272    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4cd712-dockercfg-5vclc\""
Sep 18 20:24:04.625239 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.625213    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"]
Sep 18 20:24:04.734656 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.734621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ebe8f8-fec7-440b-b1d8-016484483db9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt\" (UID: \"97ebe8f8-fec7-440b-b1d8-016484483db9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"
Sep 18 20:24:04.734845 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.734752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n69bm\" (UniqueName: \"kubernetes.io/projected/97ebe8f8-fec7-440b-b1d8-016484483db9-kube-api-access-n69bm\") pod \"payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt\" (UID: \"97ebe8f8-fec7-440b-b1d8-016484483db9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"
Sep 18 20:24:04.835837 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.835802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n69bm\" (UniqueName: \"kubernetes.io/projected/97ebe8f8-fec7-440b-b1d8-016484483db9-kube-api-access-n69bm\") pod \"payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt\" (UID: \"97ebe8f8-fec7-440b-b1d8-016484483db9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"
Sep 18 20:24:04.836037 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.835857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ebe8f8-fec7-440b-b1d8-016484483db9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt\" (UID: \"97ebe8f8-fec7-440b-b1d8-016484483db9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"
Sep 18 20:24:04.836609 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.836584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ebe8f8-fec7-440b-b1d8-016484483db9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt\" (UID: \"97ebe8f8-fec7-440b-b1d8-016484483db9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"
Sep 18 20:24:04.844012 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.843990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n69bm\" (UniqueName: \"kubernetes.io/projected/97ebe8f8-fec7-440b-b1d8-016484483db9-kube-api-access-n69bm\") pod \"payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt\" (UID: \"97ebe8f8-fec7-440b-b1d8-016484483db9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"
Sep 18 20:24:04.920078 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:04.920045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"
Sep 18 20:24:05.194530 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:05.194501    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"]
Sep 18 20:24:05.196121 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:24:05.196079    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ebe8f8_fec7_440b_b1d8_016484483db9.slice/crio-3d2da281984711b4de392506a9d5eaec86564f8740285adeb889fe6550757e80 WatchSource:0}: Error finding container 3d2da281984711b4de392506a9d5eaec86564f8740285adeb889fe6550757e80: Status 404 returned error can't find the container with id 3d2da281984711b4de392506a9d5eaec86564f8740285adeb889fe6550757e80
Sep 18 20:24:05.910174 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:05.910139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt" event={"ID":"97ebe8f8-fec7-440b-b1d8-016484483db9","Type":"ContainerStarted","Data":"1b35ded3516558fd832d9b203969b8c3ffc02ccc5fa0d122f3d9185cb3605cb5"}
Sep 18 20:24:05.910387 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:05.910195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt" event={"ID":"97ebe8f8-fec7-440b-b1d8-016484483db9","Type":"ContainerStarted","Data":"3d2da281984711b4de392506a9d5eaec86564f8740285adeb889fe6550757e80"}
Sep 18 20:24:05.910387 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:05.910354    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"
Sep 18 20:24:05.929924 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:05.929879    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt" podStartSLOduration=1.929861994 podStartE2EDuration="1.929861994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:05.928628036 +0000 UTC m=+1647.767091989" watchObservedRunningTime="2026-09-18 20:24:05.929861994 +0000 UTC m=+1647.768325949"
Sep 18 20:24:16.918129 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:16.918091    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"
Sep 18 20:24:26.828611 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:26.828572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:27.741369 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:27.741339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:28.500155 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:28.500123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:29.273398 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:29.273361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:29.373054 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:29.373018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:30.160790 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:30.160750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:31.234685 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:31.234645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:32.129398 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:32.129366    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:33.124282 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:33.124245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:33.537705 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:33.537670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:33.926702 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:33.926614    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:34.626581 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:34.626549    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:34.860302 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:34.860266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:35.673545 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:35.673514    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:36.596103 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:36.596060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:37.407352 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:37.407313    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:38.153336 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:38.153304    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:42.152822 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:42.152781    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:42.392409 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:42.392370    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:43.258279 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:43.258244    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:48.986458 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:48.986421    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"]
Sep 18 20:24:48.987028 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:48.986621    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp" podUID="21a2904f-9895-4d39-abc2-96e69deec110" containerName="maas-api" containerID="cri-o://af6816bc3e329631c691f09a8f1c52da9052fd7953e4494fdf86a8a5e412721d" gracePeriod=30
Sep 18 20:24:49.038220 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.038185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:49.064521 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.064491    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"]
Sep 18 20:24:49.064711 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.064684    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz" podUID="cb8e9951-7309-482a-8e71-c7504575f7f5" containerName="payload-processing" containerID="cri-o://14956dc2d63982c087ff1297e412e92b0cd2896275f07c56515f917608a30e9c" gracePeriod=30
Sep 18 20:24:49.079635 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.079603    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"]
Sep 18 20:24:49.079863 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.079838    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd" podUID="2220fd64-8942-4de9-98ce-448473f3406a" containerName="payload-pre-processing" containerID="cri-o://8525a4af892b723db8f93049ccddd27817c64a7cf023a17d3f440e47e671ebbf" gracePeriod=30
Sep 18 20:24:49.129167 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.129137    2578 generic.go:356] "Generic (PLEG): container finished" podID="21a2904f-9895-4d39-abc2-96e69deec110" containerID="af6816bc3e329631c691f09a8f1c52da9052fd7953e4494fdf86a8a5e412721d" exitCode=0
Sep 18 20:24:49.129300 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.129211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp" event={"ID":"21a2904f-9895-4d39-abc2-96e69deec110","Type":"ContainerDied","Data":"af6816bc3e329631c691f09a8f1c52da9052fd7953e4494fdf86a8a5e412721d"}
Sep 18 20:24:49.136632 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.136609    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:49.236631 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.236541    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:49.315500 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.315398    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:49.350238 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.350183    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"
Sep 18 20:24:49.367835 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.367810    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:24:49.377838 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.377811    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"
Sep 18 20:24:49.439007 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.438973    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2220fd64-8942-4de9-98ce-448473f3406a-plugins-config-volume\") pod \"2220fd64-8942-4de9-98ce-448473f3406a\" (UID: \"2220fd64-8942-4de9-98ce-448473f3406a\") "
Sep 18 20:24:49.439181 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.439039    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk2m9\" (UniqueName: \"kubernetes.io/projected/cb8e9951-7309-482a-8e71-c7504575f7f5-kube-api-access-lk2m9\") pod \"cb8e9951-7309-482a-8e71-c7504575f7f5\" (UID: \"cb8e9951-7309-482a-8e71-c7504575f7f5\") "
Sep 18 20:24:49.439181 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.439059    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gncrt\" (UniqueName: \"kubernetes.io/projected/21a2904f-9895-4d39-abc2-96e69deec110-kube-api-access-gncrt\") pod \"21a2904f-9895-4d39-abc2-96e69deec110\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") "
Sep 18 20:24:49.439181 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.439083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-metrics-certs\") pod \"21a2904f-9895-4d39-abc2-96e69deec110\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") "
Sep 18 20:24:49.439181 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.439122    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-maas-api-tls\") pod \"21a2904f-9895-4d39-abc2-96e69deec110\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") "
Sep 18 20:24:49.439181 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.439137    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st28q\" (UniqueName: \"kubernetes.io/projected/2220fd64-8942-4de9-98ce-448473f3406a-kube-api-access-st28q\") pod \"2220fd64-8942-4de9-98ce-448473f3406a\" (UID: \"2220fd64-8942-4de9-98ce-448473f3406a\") "
Sep 18 20:24:49.439181 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.439166    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2904f-9895-4d39-abc2-96e69deec110-openshift-service-ca\") pod \"21a2904f-9895-4d39-abc2-96e69deec110\" (UID: \"21a2904f-9895-4d39-abc2-96e69deec110\") "
Sep 18 20:24:49.439510 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.439234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8e9951-7309-482a-8e71-c7504575f7f5-plugins-config-volume\") pod \"cb8e9951-7309-482a-8e71-c7504575f7f5\" (UID: \"cb8e9951-7309-482a-8e71-c7504575f7f5\") "
Sep 18 20:24:49.439510 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.439452    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2220fd64-8942-4de9-98ce-448473f3406a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2220fd64-8942-4de9-98ce-448473f3406a" (UID: "2220fd64-8942-4de9-98ce-448473f3406a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:24:49.439625 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.439546    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2220fd64-8942-4de9-98ce-448473f3406a-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:24:49.439897 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.439855    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a2904f-9895-4d39-abc2-96e69deec110-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21a2904f-9895-4d39-abc2-96e69deec110" (UID: "21a2904f-9895-4d39-abc2-96e69deec110"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:24:49.440057 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.439924    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8e9951-7309-482a-8e71-c7504575f7f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb8e9951-7309-482a-8e71-c7504575f7f5" (UID: "cb8e9951-7309-482a-8e71-c7504575f7f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:24:49.441567 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.441538    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a2904f-9895-4d39-abc2-96e69deec110-kube-api-access-gncrt" (OuterVolumeSpecName: "kube-api-access-gncrt") pod "21a2904f-9895-4d39-abc2-96e69deec110" (UID: "21a2904f-9895-4d39-abc2-96e69deec110"). InnerVolumeSpecName "kube-api-access-gncrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:24:49.441682 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.441602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8e9951-7309-482a-8e71-c7504575f7f5-kube-api-access-lk2m9" (OuterVolumeSpecName: "kube-api-access-lk2m9") pod "cb8e9951-7309-482a-8e71-c7504575f7f5" (UID: "cb8e9951-7309-482a-8e71-c7504575f7f5"). InnerVolumeSpecName "kube-api-access-lk2m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:24:49.441682 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.441642    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2220fd64-8942-4de9-98ce-448473f3406a-kube-api-access-st28q" (OuterVolumeSpecName: "kube-api-access-st28q") pod "2220fd64-8942-4de9-98ce-448473f3406a" (UID: "2220fd64-8942-4de9-98ce-448473f3406a"). InnerVolumeSpecName "kube-api-access-st28q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:24:49.441761 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.441740    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "21a2904f-9895-4d39-abc2-96e69deec110" (UID: "21a2904f-9895-4d39-abc2-96e69deec110"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:24:49.441796 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.441751    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21a2904f-9895-4d39-abc2-96e69deec110" (UID: "21a2904f-9895-4d39-abc2-96e69deec110"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:24:49.540399 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.540309    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk2m9\" (UniqueName: \"kubernetes.io/projected/cb8e9951-7309-482a-8e71-c7504575f7f5-kube-api-access-lk2m9\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:24:49.540399 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.540342    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gncrt\" (UniqueName: \"kubernetes.io/projected/21a2904f-9895-4d39-abc2-96e69deec110-kube-api-access-gncrt\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:24:49.540399 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.540353    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:24:49.540399 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.540361    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a2904f-9895-4d39-abc2-96e69deec110-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:24:49.540399 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.540371    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st28q\" (UniqueName: \"kubernetes.io/projected/2220fd64-8942-4de9-98ce-448473f3406a-kube-api-access-st28q\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:24:49.540399 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.540378    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2904f-9895-4d39-abc2-96e69deec110-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:24:49.540399 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.540387    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8e9951-7309-482a-8e71-c7504575f7f5-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:24:49.658191 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.658152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:49.757890 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:49.757860    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:50.135681 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.135648    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"
Sep 18 20:24:50.136153 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.135650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp" event={"ID":"21a2904f-9895-4d39-abc2-96e69deec110","Type":"ContainerDied","Data":"7dbec74db9fdc69ecd09f030c46603d2695fa437c656f039778419f00d90fdd3"}
Sep 18 20:24:50.136153 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.135816    2578 scope.go:117] "RemoveContainer" containerID="af6816bc3e329631c691f09a8f1c52da9052fd7953e4494fdf86a8a5e412721d"
Sep 18 20:24:50.137299 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.137102    2578 generic.go:356] "Generic (PLEG): container finished" podID="2220fd64-8942-4de9-98ce-448473f3406a" containerID="8525a4af892b723db8f93049ccddd27817c64a7cf023a17d3f440e47e671ebbf" exitCode=0
Sep 18 20:24:50.137299 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.137143    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"
Sep 18 20:24:50.137299 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.137174    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd" event={"ID":"2220fd64-8942-4de9-98ce-448473f3406a","Type":"ContainerDied","Data":"8525a4af892b723db8f93049ccddd27817c64a7cf023a17d3f440e47e671ebbf"}
Sep 18 20:24:50.137299 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.137217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd" event={"ID":"2220fd64-8942-4de9-98ce-448473f3406a","Type":"ContainerDied","Data":"6f76aa9ecb00b828e60e9440d0efef41e8a08658cf43668643baece51c9b43b1"}
Sep 18 20:24:50.138724 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.138705    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb8e9951-7309-482a-8e71-c7504575f7f5" containerID="14956dc2d63982c087ff1297e412e92b0cd2896275f07c56515f917608a30e9c" exitCode=0
Sep 18 20:24:50.138876 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.138752    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz" event={"ID":"cb8e9951-7309-482a-8e71-c7504575f7f5","Type":"ContainerDied","Data":"14956dc2d63982c087ff1297e412e92b0cd2896275f07c56515f917608a30e9c"}
Sep 18 20:24:50.138876 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.138783    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"
Sep 18 20:24:50.138876 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.138804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz" event={"ID":"cb8e9951-7309-482a-8e71-c7504575f7f5","Type":"ContainerDied","Data":"5dc1b07b5cbcc78c1881a301703e466b4b47a0a462834b201b59b4d7c0b14933"}
Sep 18 20:24:50.147301 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.147283    2578 scope.go:117] "RemoveContainer" containerID="8525a4af892b723db8f93049ccddd27817c64a7cf023a17d3f440e47e671ebbf"
Sep 18 20:24:50.155604 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.155583    2578 scope.go:117] "RemoveContainer" containerID="8525a4af892b723db8f93049ccddd27817c64a7cf023a17d3f440e47e671ebbf"
Sep 18 20:24:50.155884 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:24:50.155865    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8525a4af892b723db8f93049ccddd27817c64a7cf023a17d3f440e47e671ebbf\": container with ID starting with 8525a4af892b723db8f93049ccddd27817c64a7cf023a17d3f440e47e671ebbf not found: ID does not exist" containerID="8525a4af892b723db8f93049ccddd27817c64a7cf023a17d3f440e47e671ebbf"
Sep 18 20:24:50.155982 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.155891    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8525a4af892b723db8f93049ccddd27817c64a7cf023a17d3f440e47e671ebbf"} err="failed to get container status \"8525a4af892b723db8f93049ccddd27817c64a7cf023a17d3f440e47e671ebbf\": rpc error: code = NotFound desc = could not find container \"8525a4af892b723db8f93049ccddd27817c64a7cf023a17d3f440e47e671ebbf\": container with ID starting with 8525a4af892b723db8f93049ccddd27817c64a7cf023a17d3f440e47e671ebbf not found: ID does not exist"
Sep 18 20:24:50.155982 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.155909    2578 scope.go:117] "RemoveContainer" containerID="14956dc2d63982c087ff1297e412e92b0cd2896275f07c56515f917608a30e9c"
Sep 18 20:24:50.162189 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.162167    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"]
Sep 18 20:24:50.164087 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.164061    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-69f1ba-56887fcbb-2xjjp"]
Sep 18 20:24:50.164640 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.164622    2578 scope.go:117] "RemoveContainer" containerID="14956dc2d63982c087ff1297e412e92b0cd2896275f07c56515f917608a30e9c"
Sep 18 20:24:50.164915 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:24:50.164895    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14956dc2d63982c087ff1297e412e92b0cd2896275f07c56515f917608a30e9c\": container with ID starting with 14956dc2d63982c087ff1297e412e92b0cd2896275f07c56515f917608a30e9c not found: ID does not exist" containerID="14956dc2d63982c087ff1297e412e92b0cd2896275f07c56515f917608a30e9c"
Sep 18 20:24:50.164990 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.164919    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14956dc2d63982c087ff1297e412e92b0cd2896275f07c56515f917608a30e9c"} err="failed to get container status \"14956dc2d63982c087ff1297e412e92b0cd2896275f07c56515f917608a30e9c\": rpc error: code = NotFound desc = could not find container \"14956dc2d63982c087ff1297e412e92b0cd2896275f07c56515f917608a30e9c\": container with ID starting with 14956dc2d63982c087ff1297e412e92b0cd2896275f07c56515f917608a30e9c not found: ID does not exist"
Sep 18 20:24:50.173545 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.173525    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"]
Sep 18 20:24:50.179867 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.179846    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-69f1ba-59b5d6fbb6-xbdpd"]
Sep 18 20:24:50.192324 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.192302    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"]
Sep 18 20:24:50.194703 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.194684    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-69f1ba-7b76fd84d5-n6mjz"]
Sep 18 20:24:50.319629 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.319555    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21a2904f-9895-4d39-abc2-96e69deec110" path="/var/lib/kubelet/pods/21a2904f-9895-4d39-abc2-96e69deec110/volumes"
Sep 18 20:24:50.319929 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.319916    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2220fd64-8942-4de9-98ce-448473f3406a" path="/var/lib/kubelet/pods/2220fd64-8942-4de9-98ce-448473f3406a/volumes"
Sep 18 20:24:50.320289 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.320276    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb8e9951-7309-482a-8e71-c7504575f7f5" path="/var/lib/kubelet/pods/cb8e9951-7309-482a-8e71-c7504575f7f5/volumes"
Sep 18 20:24:50.530515 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:50.530481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:51.624701 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:51.624663    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:52.472673 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:52.472590    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:53.384915 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:53.384878    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:54.414516 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:54.414488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:55.740699 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:55.740661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:55.841563 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:55.841526    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:56.014407 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:56.014326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:56.498389 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:56.498349    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:57.351658 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:57.351628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:58.191305 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:58.191271    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:24:59.284192 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:24:59.284153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:00.118319 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:00.118283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:01.035129 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:01.035092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:01.797054 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:01.797018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:11.983490 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:11.983455    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:12.019006 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.018973    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"]
Sep 18 20:25:12.019235 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.019197    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt" podUID="97ebe8f8-fec7-440b-b1d8-016484483db9" containerName="payload-pre-processing" containerID="cri-o://1b35ded3516558fd832d9b203969b8c3ffc02ccc5fa0d122f3d9185cb3605cb5" gracePeriod=30
Sep 18 20:25:12.085117 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.085086    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:12.185176 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.185127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:12.250450 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.250372    2578 generic.go:356] "Generic (PLEG): container finished" podID="97ebe8f8-fec7-440b-b1d8-016484483db9" containerID="1b35ded3516558fd832d9b203969b8c3ffc02ccc5fa0d122f3d9185cb3605cb5" exitCode=0
Sep 18 20:25:12.250450 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.250433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt" event={"ID":"97ebe8f8-fec7-440b-b1d8-016484483db9","Type":"ContainerDied","Data":"1b35ded3516558fd832d9b203969b8c3ffc02ccc5fa0d122f3d9185cb3605cb5"}
Sep 18 20:25:12.274288 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.274261    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"
Sep 18 20:25:12.284137 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.284109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:12.328909 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.328878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ebe8f8-fec7-440b-b1d8-016484483db9-plugins-config-volume\") pod \"97ebe8f8-fec7-440b-b1d8-016484483db9\" (UID: \"97ebe8f8-fec7-440b-b1d8-016484483db9\") "
Sep 18 20:25:12.328909 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.328910    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n69bm\" (UniqueName: \"kubernetes.io/projected/97ebe8f8-fec7-440b-b1d8-016484483db9-kube-api-access-n69bm\") pod \"97ebe8f8-fec7-440b-b1d8-016484483db9\" (UID: \"97ebe8f8-fec7-440b-b1d8-016484483db9\") "
Sep 18 20:25:12.329342 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.329315    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ebe8f8-fec7-440b-b1d8-016484483db9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "97ebe8f8-fec7-440b-b1d8-016484483db9" (UID: "97ebe8f8-fec7-440b-b1d8-016484483db9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:12.332242 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.332210    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ebe8f8-fec7-440b-b1d8-016484483db9-kube-api-access-n69bm" (OuterVolumeSpecName: "kube-api-access-n69bm") pod "97ebe8f8-fec7-440b-b1d8-016484483db9" (UID: "97ebe8f8-fec7-440b-b1d8-016484483db9"). InnerVolumeSpecName "kube-api-access-n69bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:12.430180 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.430150    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ebe8f8-fec7-440b-b1d8-016484483db9-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:25:12.430180 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.430176    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n69bm\" (UniqueName: \"kubernetes.io/projected/97ebe8f8-fec7-440b-b1d8-016484483db9-kube-api-access-n69bm\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:25:12.570573 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.570485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:12.671113 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:12.671078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:13.256883 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:13.256856    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"
Sep 18 20:25:13.257291 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:13.256857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt" event={"ID":"97ebe8f8-fec7-440b-b1d8-016484483db9","Type":"ContainerDied","Data":"3d2da281984711b4de392506a9d5eaec86564f8740285adeb889fe6550757e80"}
Sep 18 20:25:13.257291 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:13.257007    2578 scope.go:117] "RemoveContainer" containerID="1b35ded3516558fd832d9b203969b8c3ffc02ccc5fa0d122f3d9185cb3605cb5"
Sep 18 20:25:13.279181 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:13.279151    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"]
Sep 18 20:25:13.291914 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:13.287418    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4cd712-74cbd79c74-k6ggt"]
Sep 18 20:25:13.421281 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:13.421242    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:14.303317 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:14.303281    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:14.317885 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:14.317858    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97ebe8f8-fec7-440b-b1d8-016484483db9" path="/var/lib/kubelet/pods/97ebe8f8-fec7-440b-b1d8-016484483db9/volumes"
Sep 18 20:25:15.039425 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.039391    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:15.615777 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.615741    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"]
Sep 18 20:25:15.616168 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.616128    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2220fd64-8942-4de9-98ce-448473f3406a" containerName="payload-pre-processing"
Sep 18 20:25:15.616168 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.616141    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2220fd64-8942-4de9-98ce-448473f3406a" containerName="payload-pre-processing"
Sep 18 20:25:15.616168 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.616151    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ebe8f8-fec7-440b-b1d8-016484483db9" containerName="payload-pre-processing"
Sep 18 20:25:15.616168 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.616157    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ebe8f8-fec7-440b-b1d8-016484483db9" containerName="payload-pre-processing"
Sep 18 20:25:15.616305 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.616180    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8e9951-7309-482a-8e71-c7504575f7f5" containerName="payload-processing"
Sep 18 20:25:15.616305 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.616186    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8e9951-7309-482a-8e71-c7504575f7f5" containerName="payload-processing"
Sep 18 20:25:15.616305 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.616198    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a2904f-9895-4d39-abc2-96e69deec110" containerName="maas-api"
Sep 18 20:25:15.616305 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.616203    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a2904f-9895-4d39-abc2-96e69deec110" containerName="maas-api"
Sep 18 20:25:15.616305 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.616258    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a2904f-9895-4d39-abc2-96e69deec110" containerName="maas-api"
Sep 18 20:25:15.616305 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.616267    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2220fd64-8942-4de9-98ce-448473f3406a" containerName="payload-pre-processing"
Sep 18 20:25:15.616305 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.616274    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8e9951-7309-482a-8e71-c7504575f7f5" containerName="payload-processing"
Sep 18 20:25:15.616305 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.616282    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ebe8f8-fec7-440b-b1d8-016484483db9" containerName="payload-pre-processing"
Sep 18 20:25:15.620767 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.620743    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.623325 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.623302    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-81f19d\"/\"kube-root-ca.crt\""
Sep 18 20:25:15.623972 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.623950    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-81f19d\"/\"sub-test-model-81f19d-kserve-self-signed-certs\""
Sep 18 20:25:15.624090 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.624071    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-81f19d\"/\"default-dockercfg-kwxkf\""
Sep 18 20:25:15.624332 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.624314    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-81f19d\"/\"openshift-service-ca.crt\""
Sep 18 20:25:15.632193 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.632173    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"]
Sep 18 20:25:15.654551 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.654530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-model-cache\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.654638 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.654558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-dshm\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.654638 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.654585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-home\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.654638 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.654604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89ftl\" (UniqueName: \"kubernetes.io/projected/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-kube-api-access-89ftl\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.654764 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.654702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-tmp-dir\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.654764 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.654735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-tls-certs\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.658232 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.658198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:15.755426 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.755393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-model-cache\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.755594 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.755431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-dshm\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.755594 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.755557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-home\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.755594 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.755589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89ftl\" (UniqueName: \"kubernetes.io/projected/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-kube-api-access-89ftl\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.755756 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.755623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-tmp-dir\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.755756 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.755652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-tls-certs\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.755866 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.755777    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-model-cache\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.755956 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.755912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-home\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.756153 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.756133    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-tmp-dir\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.757820 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.757796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-dshm\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.758139 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.758122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-tls-certs\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.764572 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.764551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89ftl\" (UniqueName: \"kubernetes.io/projected/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-kube-api-access-89ftl\") pod \"sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") " pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:15.830287 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.830254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:15.930461 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:15.930373    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:16.205455 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:16.205305    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"]
Sep 18 20:25:16.208059 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:25:16.208027    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fc6684b_d2f4_4fc6_aa7a_11efb38d5d68.slice/crio-81c3368fc30fe40c6c9bb4427831d7eb21e37b201e0108b9df65cf672a916d3d WatchSource:0}: Error finding container 81c3368fc30fe40c6c9bb4427831d7eb21e37b201e0108b9df65cf672a916d3d: Status 404 returned error can't find the container with id 81c3368fc30fe40c6c9bb4427831d7eb21e37b201e0108b9df65cf672a916d3d
Sep 18 20:25:16.274539 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:16.274516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j" event={"ID":"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68","Type":"ContainerStarted","Data":"ef3c9ac0a004a455383405ea499e6480d8e819e9b05aab9db0f4aacd99ca2cab"}
Sep 18 20:25:16.274653 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:16.274553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j" event={"ID":"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68","Type":"ContainerStarted","Data":"81c3368fc30fe40c6c9bb4427831d7eb21e37b201e0108b9df65cf672a916d3d"}
Sep 18 20:25:16.300031 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:16.299971    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j" podStartSLOduration=1.299952206 podStartE2EDuration="1.299952206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:25:16.297111417 +0000 UTC m=+1718.135575370" watchObservedRunningTime="2026-09-18 20:25:16.299952206 +0000 UTC m=+1718.138416155"
Sep 18 20:25:19.113137 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:19.113094    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:19.248599 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:19.248560    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:19.406842 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:19.406753    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:19.883664 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:19.883628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:20.828688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:20.828658    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:21.953658 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:21.953625    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:22.767003 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:22.766964    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:23.628209 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:23.628176    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:24.545427 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:24.545391    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:25.931331 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:25.931293    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:25.931824 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:25.931355    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:25.943757 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:25.943728    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:26.334173 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:26.334144    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:25:30.448734 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.448693    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"]
Sep 18 20:25:30.449257 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.448969    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj" podUID="7fd8789f-ad30-4250-9127-5992c719f1b1" containerName="maas-api" containerID="cri-o://abf14a1a8d6e96f9b87643accfcdd9654277cbb6b06c64ad7194bbac95b138cf" gracePeriod=30
Sep 18 20:25:30.488696 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.488661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:30.520758 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.520727    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"]
Sep 18 20:25:30.521037 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.521009    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm" podUID="a7769618-6fa3-4145-9912-4ae09261182a" containerName="payload-pre-processing" containerID="cri-o://20b56923672b0e534d0087f34b3153e1f9770ed294a081a601efe44fd5eff820" gracePeriod=30
Sep 18 20:25:30.591280 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.591243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:30.689923 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.689887    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:30.759108 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.759073    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:30.766557 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.766533    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:25:30.792309 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.792283    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"
Sep 18 20:25:30.884992 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.884953    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt26p\" (UniqueName: \"kubernetes.io/projected/a7769618-6fa3-4145-9912-4ae09261182a-kube-api-access-lt26p\") pod \"a7769618-6fa3-4145-9912-4ae09261182a\" (UID: \"a7769618-6fa3-4145-9912-4ae09261182a\") "
Sep 18 20:25:30.885168 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.885010    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-metrics-certs\") pod \"7fd8789f-ad30-4250-9127-5992c719f1b1\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") "
Sep 18 20:25:30.885235 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.885169    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7769618-6fa3-4145-9912-4ae09261182a-plugins-config-volume\") pod \"a7769618-6fa3-4145-9912-4ae09261182a\" (UID: \"a7769618-6fa3-4145-9912-4ae09261182a\") "
Sep 18 20:25:30.885296 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.885234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd8789f-ad30-4250-9127-5992c719f1b1-openshift-service-ca\") pod \"7fd8789f-ad30-4250-9127-5992c719f1b1\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") "
Sep 18 20:25:30.885383 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.885361    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-maas-api-tls\") pod \"7fd8789f-ad30-4250-9127-5992c719f1b1\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") "
Sep 18 20:25:30.885597 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.885565    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxcqt\" (UniqueName: \"kubernetes.io/projected/7fd8789f-ad30-4250-9127-5992c719f1b1-kube-api-access-zxcqt\") pod \"7fd8789f-ad30-4250-9127-5992c719f1b1\" (UID: \"7fd8789f-ad30-4250-9127-5992c719f1b1\") "
Sep 18 20:25:30.885597 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.885574    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd8789f-ad30-4250-9127-5992c719f1b1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7fd8789f-ad30-4250-9127-5992c719f1b1" (UID: "7fd8789f-ad30-4250-9127-5992c719f1b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:30.885597 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.885583    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7769618-6fa3-4145-9912-4ae09261182a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7769618-6fa3-4145-9912-4ae09261182a" (UID: "a7769618-6fa3-4145-9912-4ae09261182a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:30.885871 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.885852    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd8789f-ad30-4250-9127-5992c719f1b1-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:25:30.885923 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.885878    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7769618-6fa3-4145-9912-4ae09261182a-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:25:30.887272 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.887241    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7fd8789f-ad30-4250-9127-5992c719f1b1" (UID: "7fd8789f-ad30-4250-9127-5992c719f1b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:25:30.887272 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.887247    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7769618-6fa3-4145-9912-4ae09261182a-kube-api-access-lt26p" (OuterVolumeSpecName: "kube-api-access-lt26p") pod "a7769618-6fa3-4145-9912-4ae09261182a" (UID: "a7769618-6fa3-4145-9912-4ae09261182a"). InnerVolumeSpecName "kube-api-access-lt26p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:30.887436 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.887301    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7fd8789f-ad30-4250-9127-5992c719f1b1" (UID: "7fd8789f-ad30-4250-9127-5992c719f1b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:25:30.887484 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.887469    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd8789f-ad30-4250-9127-5992c719f1b1-kube-api-access-zxcqt" (OuterVolumeSpecName: "kube-api-access-zxcqt") pod "7fd8789f-ad30-4250-9127-5992c719f1b1" (UID: "7fd8789f-ad30-4250-9127-5992c719f1b1"). InnerVolumeSpecName "kube-api-access-zxcqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:30.987251 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.987215    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:25:30.987251 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.987247    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxcqt\" (UniqueName: \"kubernetes.io/projected/7fd8789f-ad30-4250-9127-5992c719f1b1-kube-api-access-zxcqt\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:25:30.987251 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.987259    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt26p\" (UniqueName: \"kubernetes.io/projected/a7769618-6fa3-4145-9912-4ae09261182a-kube-api-access-lt26p\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:25:30.987500 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:30.987270    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fd8789f-ad30-4250-9127-5992c719f1b1-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:25:31.079688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.079648    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:31.185357 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.185324    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:31.349007 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.348903    2578 generic.go:356] "Generic (PLEG): container finished" podID="7fd8789f-ad30-4250-9127-5992c719f1b1" containerID="abf14a1a8d6e96f9b87643accfcdd9654277cbb6b06c64ad7194bbac95b138cf" exitCode=0
Sep 18 20:25:31.349007 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.348959    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"
Sep 18 20:25:31.349007 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.348997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj" event={"ID":"7fd8789f-ad30-4250-9127-5992c719f1b1","Type":"ContainerDied","Data":"abf14a1a8d6e96f9b87643accfcdd9654277cbb6b06c64ad7194bbac95b138cf"}
Sep 18 20:25:31.349309 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.349044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj" event={"ID":"7fd8789f-ad30-4250-9127-5992c719f1b1","Type":"ContainerDied","Data":"12ec601f402583b44e8aa8246813a0bcb82e948a496a60b7fb6d890d653d763a"}
Sep 18 20:25:31.349309 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.349073    2578 scope.go:117] "RemoveContainer" containerID="abf14a1a8d6e96f9b87643accfcdd9654277cbb6b06c64ad7194bbac95b138cf"
Sep 18 20:25:31.350354 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.350329    2578 generic.go:356] "Generic (PLEG): container finished" podID="a7769618-6fa3-4145-9912-4ae09261182a" containerID="20b56923672b0e534d0087f34b3153e1f9770ed294a081a601efe44fd5eff820" exitCode=0
Sep 18 20:25:31.350459 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.350376    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"
Sep 18 20:25:31.350459 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.350406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm" event={"ID":"a7769618-6fa3-4145-9912-4ae09261182a","Type":"ContainerDied","Data":"20b56923672b0e534d0087f34b3153e1f9770ed294a081a601efe44fd5eff820"}
Sep 18 20:25:31.350459 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.350439    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm" event={"ID":"a7769618-6fa3-4145-9912-4ae09261182a","Type":"ContainerDied","Data":"65ca2862d1584b03dc4d027bf21655bcc056340ff40bf9cd5fdca10390ccf716"}
Sep 18 20:25:31.359240 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.359223    2578 scope.go:117] "RemoveContainer" containerID="abf14a1a8d6e96f9b87643accfcdd9654277cbb6b06c64ad7194bbac95b138cf"
Sep 18 20:25:31.359549 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:25:31.359530    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf14a1a8d6e96f9b87643accfcdd9654277cbb6b06c64ad7194bbac95b138cf\": container with ID starting with abf14a1a8d6e96f9b87643accfcdd9654277cbb6b06c64ad7194bbac95b138cf not found: ID does not exist" containerID="abf14a1a8d6e96f9b87643accfcdd9654277cbb6b06c64ad7194bbac95b138cf"
Sep 18 20:25:31.359603 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.359558    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf14a1a8d6e96f9b87643accfcdd9654277cbb6b06c64ad7194bbac95b138cf"} err="failed to get container status \"abf14a1a8d6e96f9b87643accfcdd9654277cbb6b06c64ad7194bbac95b138cf\": rpc error: code = NotFound desc = could not find container \"abf14a1a8d6e96f9b87643accfcdd9654277cbb6b06c64ad7194bbac95b138cf\": container with ID starting with abf14a1a8d6e96f9b87643accfcdd9654277cbb6b06c64ad7194bbac95b138cf not found: ID does not exist"
Sep 18 20:25:31.359603 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.359577    2578 scope.go:117] "RemoveContainer" containerID="20b56923672b0e534d0087f34b3153e1f9770ed294a081a601efe44fd5eff820"
Sep 18 20:25:31.367427 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.367410    2578 scope.go:117] "RemoveContainer" containerID="20b56923672b0e534d0087f34b3153e1f9770ed294a081a601efe44fd5eff820"
Sep 18 20:25:31.367691 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:25:31.367673    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b56923672b0e534d0087f34b3153e1f9770ed294a081a601efe44fd5eff820\": container with ID starting with 20b56923672b0e534d0087f34b3153e1f9770ed294a081a601efe44fd5eff820 not found: ID does not exist" containerID="20b56923672b0e534d0087f34b3153e1f9770ed294a081a601efe44fd5eff820"
Sep 18 20:25:31.367741 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.367697    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b56923672b0e534d0087f34b3153e1f9770ed294a081a601efe44fd5eff820"} err="failed to get container status \"20b56923672b0e534d0087f34b3153e1f9770ed294a081a601efe44fd5eff820\": rpc error: code = NotFound desc = could not find container \"20b56923672b0e534d0087f34b3153e1f9770ed294a081a601efe44fd5eff820\": container with ID starting with 20b56923672b0e534d0087f34b3153e1f9770ed294a081a601efe44fd5eff820 not found: ID does not exist"
Sep 18 20:25:31.382578 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.382557    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"]
Sep 18 20:25:31.388056 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.388035    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5319c6-76488b4465-8lwqj"]
Sep 18 20:25:31.400015 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.399994    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"]
Sep 18 20:25:31.402876 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:31.402857    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5319c6-7d59bcf4fd-j2cnm"]
Sep 18 20:25:32.061752 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:32.061719    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:32.317873 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:32.317797    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fd8789f-ad30-4250-9127-5992c719f1b1" path="/var/lib/kubelet/pods/7fd8789f-ad30-4250-9127-5992c719f1b1/volumes"
Sep 18 20:25:32.318280 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:32.318261    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7769618-6fa3-4145-9912-4ae09261182a" path="/var/lib/kubelet/pods/a7769618-6fa3-4145-9912-4ae09261182a/volumes"
Sep 18 20:25:32.861375 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:32.861337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:33.629677 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:33.629644    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:37.095780 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:37.095748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:37.198319 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:37.198272    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:37.369981 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:37.369881    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:37.907132 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:37.907095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:38.006376 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:38.006338    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:38.931743 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:38.931708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:39.833870 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:39.833837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:40.731475 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:40.731423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:41.630926 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:41.630883    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:48.496677 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:48.496617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:48.539240 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:48.539212    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"]
Sep 18 20:25:48.539421 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:48.539400    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t" podUID="bae2c0d9-df61-4a95-a80e-d375a75f9722" containerName="payload-pre-processing" containerID="cri-o://1f22421567a5f160a71c0747d5d05f4260b0c23f0747282b1a5c9ac6e9b7d631" gracePeriod=30
Sep 18 20:25:48.594867 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:48.594828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:48.694352 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:48.694318    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:48.791894 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:48.791866    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:48.796611 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:48.796589    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"
Sep 18 20:25:48.947035 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:48.947005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae2c0d9-df61-4a95-a80e-d375a75f9722-plugins-config-volume\") pod \"bae2c0d9-df61-4a95-a80e-d375a75f9722\" (UID: \"bae2c0d9-df61-4a95-a80e-d375a75f9722\") "
Sep 18 20:25:48.947212 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:48.947043    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbjww\" (UniqueName: \"kubernetes.io/projected/bae2c0d9-df61-4a95-a80e-d375a75f9722-kube-api-access-dbjww\") pod \"bae2c0d9-df61-4a95-a80e-d375a75f9722\" (UID: \"bae2c0d9-df61-4a95-a80e-d375a75f9722\") "
Sep 18 20:25:48.947391 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:48.947361    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae2c0d9-df61-4a95-a80e-d375a75f9722-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bae2c0d9-df61-4a95-a80e-d375a75f9722" (UID: "bae2c0d9-df61-4a95-a80e-d375a75f9722"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:48.949243 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:48.949222    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae2c0d9-df61-4a95-a80e-d375a75f9722-kube-api-access-dbjww" (OuterVolumeSpecName: "kube-api-access-dbjww") pod "bae2c0d9-df61-4a95-a80e-d375a75f9722" (UID: "bae2c0d9-df61-4a95-a80e-d375a75f9722"). InnerVolumeSpecName "kube-api-access-dbjww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:49.048561 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:49.048486    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae2c0d9-df61-4a95-a80e-d375a75f9722-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:25:49.048561 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:49.048512    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbjww\" (UniqueName: \"kubernetes.io/projected/bae2c0d9-df61-4a95-a80e-d375a75f9722-kube-api-access-dbjww\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:25:49.143190 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:49.143156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:49.260854 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:49.260819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:49.438336 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:49.438245    2578 generic.go:356] "Generic (PLEG): container finished" podID="bae2c0d9-df61-4a95-a80e-d375a75f9722" containerID="1f22421567a5f160a71c0747d5d05f4260b0c23f0747282b1a5c9ac6e9b7d631" exitCode=0
Sep 18 20:25:49.438336 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:49.438292    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"
Sep 18 20:25:49.438529 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:49.438334    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t" event={"ID":"bae2c0d9-df61-4a95-a80e-d375a75f9722","Type":"ContainerDied","Data":"1f22421567a5f160a71c0747d5d05f4260b0c23f0747282b1a5c9ac6e9b7d631"}
Sep 18 20:25:49.438529 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:49.438391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t" event={"ID":"bae2c0d9-df61-4a95-a80e-d375a75f9722","Type":"ContainerDied","Data":"0b253fa2dad633d3bffc674c5c566cb013c62d26a1985311a058544ab45040ec"}
Sep 18 20:25:49.438529 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:49.438414    2578 scope.go:117] "RemoveContainer" containerID="1f22421567a5f160a71c0747d5d05f4260b0c23f0747282b1a5c9ac6e9b7d631"
Sep 18 20:25:49.447804 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:49.447784    2578 scope.go:117] "RemoveContainer" containerID="1f22421567a5f160a71c0747d5d05f4260b0c23f0747282b1a5c9ac6e9b7d631"
Sep 18 20:25:49.448148 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:25:49.448124    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f22421567a5f160a71c0747d5d05f4260b0c23f0747282b1a5c9ac6e9b7d631\": container with ID starting with 1f22421567a5f160a71c0747d5d05f4260b0c23f0747282b1a5c9ac6e9b7d631 not found: ID does not exist" containerID="1f22421567a5f160a71c0747d5d05f4260b0c23f0747282b1a5c9ac6e9b7d631"
Sep 18 20:25:49.448241 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:49.448155    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f22421567a5f160a71c0747d5d05f4260b0c23f0747282b1a5c9ac6e9b7d631"} err="failed to get container status \"1f22421567a5f160a71c0747d5d05f4260b0c23f0747282b1a5c9ac6e9b7d631\": rpc error: code = NotFound desc = could not find container \"1f22421567a5f160a71c0747d5d05f4260b0c23f0747282b1a5c9ac6e9b7d631\": container with ID starting with 1f22421567a5f160a71c0747d5d05f4260b0c23f0747282b1a5c9ac6e9b7d631 not found: ID does not exist"
Sep 18 20:25:49.461568 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:49.461531    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"]
Sep 18 20:25:49.471281 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:49.471250    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c5206e-5bc4ddbd58-6f82t"]
Sep 18 20:25:50.180107 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:50.180068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:50.320310 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:50.320272    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bae2c0d9-df61-4a95-a80e-d375a75f9722" path="/var/lib/kubelet/pods/bae2c0d9-df61-4a95-a80e-d375a75f9722/volumes"
Sep 18 20:25:51.139221 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:51.139183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:55.037838 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:55.037806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:55.139409 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:55.139373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:25:55.284378 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:25:55.284341    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:04.977596 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:04.977553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:05.852701 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:05.852668    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"]
Sep 18 20:26:05.874465 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:05.874437    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:17.970132 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:17.970090    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:26:20.871165 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:20.871122    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j" podUID="2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68" containerName="main" containerID="cri-o://ef3c9ac0a004a455383405ea499e6480d8e819e9b05aab9db0f4aacd99ca2cab" gracePeriod=45
Sep 18 20:26:21.116575 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.116547    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:26:21.225144 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.225112    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89ftl\" (UniqueName: \"kubernetes.io/projected/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-kube-api-access-89ftl\") pod \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") "
Sep 18 20:26:21.225302 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.225162    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-tmp-dir\") pod \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") "
Sep 18 20:26:21.225302 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.225187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-tls-certs\") pod \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") "
Sep 18 20:26:21.225302 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.225232    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-model-cache\") pod \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") "
Sep 18 20:26:21.225302 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.225246    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-home\") pod \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") "
Sep 18 20:26:21.225302 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.225276    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-dshm\") pod \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\" (UID: \"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68\") "
Sep 18 20:26:21.225567 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.225428    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68" (UID: "2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:21.225628 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.225595    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68" (UID: "2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:21.225628 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.225605    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-home" (OuterVolumeSpecName: "home") pod "2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68" (UID: "2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:21.227370 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.227343    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-kube-api-access-89ftl" (OuterVolumeSpecName: "kube-api-access-89ftl") pod "2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68" (UID: "2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68"). InnerVolumeSpecName "kube-api-access-89ftl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:26:21.227469 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.227403    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68" (UID: "2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:26:21.227521 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.227478    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-dshm" (OuterVolumeSpecName: "dshm") pod "2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68" (UID: "2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:21.326855 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.326817    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-tmp-dir\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:26:21.326855 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.326849    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-tls-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:26:21.326855 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.326858    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-model-cache\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:26:21.327111 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.326866    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-home\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:26:21.327111 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.326874    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-dshm\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:26:21.327111 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.326883    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89ftl\" (UniqueName: \"kubernetes.io/projected/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68-kube-api-access-89ftl\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:26:21.595443 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.595350    2578 generic.go:356] "Generic (PLEG): container finished" podID="2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68" containerID="ef3c9ac0a004a455383405ea499e6480d8e819e9b05aab9db0f4aacd99ca2cab" exitCode=0
Sep 18 20:26:21.595443 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.595405    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j" event={"ID":"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68","Type":"ContainerDied","Data":"ef3c9ac0a004a455383405ea499e6480d8e819e9b05aab9db0f4aacd99ca2cab"}
Sep 18 20:26:21.595443 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.595416    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"
Sep 18 20:26:21.595443 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.595444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j" event={"ID":"2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68","Type":"ContainerDied","Data":"81c3368fc30fe40c6c9bb4427831d7eb21e37b201e0108b9df65cf672a916d3d"}
Sep 18 20:26:21.595761 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.595466    2578 scope.go:117] "RemoveContainer" containerID="ef3c9ac0a004a455383405ea499e6480d8e819e9b05aab9db0f4aacd99ca2cab"
Sep 18 20:26:21.604325 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.604309    2578 scope.go:117] "RemoveContainer" containerID="ef3c9ac0a004a455383405ea499e6480d8e819e9b05aab9db0f4aacd99ca2cab"
Sep 18 20:26:21.604621 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:26:21.604603    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3c9ac0a004a455383405ea499e6480d8e819e9b05aab9db0f4aacd99ca2cab\": container with ID starting with ef3c9ac0a004a455383405ea499e6480d8e819e9b05aab9db0f4aacd99ca2cab not found: ID does not exist" containerID="ef3c9ac0a004a455383405ea499e6480d8e819e9b05aab9db0f4aacd99ca2cab"
Sep 18 20:26:21.604683 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.604634    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3c9ac0a004a455383405ea499e6480d8e819e9b05aab9db0f4aacd99ca2cab"} err="failed to get container status \"ef3c9ac0a004a455383405ea499e6480d8e819e9b05aab9db0f4aacd99ca2cab\": rpc error: code = NotFound desc = could not find container \"ef3c9ac0a004a455383405ea499e6480d8e819e9b05aab9db0f4aacd99ca2cab\": container with ID starting with ef3c9ac0a004a455383405ea499e6480d8e819e9b05aab9db0f4aacd99ca2cab not found: ID does not exist"
Sep 18 20:26:21.619412 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.619389    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"]
Sep 18 20:26:21.625784 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:21.625762    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-81f19d/sub-test-model-81f19d-kserve-67fb6d4f7c-2d56j"]
Sep 18 20:26:22.318489 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:22.318404    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68" path="/var/lib/kubelet/pods/2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68/volumes"
Sep 18 20:26:50.733064 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:26:50.733025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:15.835396 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:15.835361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:18.994975 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:18.994928    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:26.990001 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:26.989965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:28.216927 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:28.216888    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:28.639826 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:28.639747    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:29.023796 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:29.023760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:29.151605 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:29.151570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:31.521924 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:31.521893    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:31.606876 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:31.606842    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:31.961129 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:31.961042    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:32.610094 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:32.610056    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:32.695377 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:32.695339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:33.039472 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:33.039436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:34.002779 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:34.002740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:35.023164 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.023127    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"]
Sep 18 20:27:35.023618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.023470    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68" containerName="main"
Sep 18 20:27:35.023618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.023485    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68" containerName="main"
Sep 18 20:27:35.023618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.023496    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7769618-6fa3-4145-9912-4ae09261182a" containerName="payload-pre-processing"
Sep 18 20:27:35.023618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.023501    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7769618-6fa3-4145-9912-4ae09261182a" containerName="payload-pre-processing"
Sep 18 20:27:35.023618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.023510    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd8789f-ad30-4250-9127-5992c719f1b1" containerName="maas-api"
Sep 18 20:27:35.023618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.023515    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd8789f-ad30-4250-9127-5992c719f1b1" containerName="maas-api"
Sep 18 20:27:35.023618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.023523    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bae2c0d9-df61-4a95-a80e-d375a75f9722" containerName="payload-pre-processing"
Sep 18 20:27:35.023618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.023528    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bae2c0d9-df61-4a95-a80e-d375a75f9722" containerName="payload-pre-processing"
Sep 18 20:27:35.023618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.023590    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7769618-6fa3-4145-9912-4ae09261182a" containerName="payload-pre-processing"
Sep 18 20:27:35.023618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.023597    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bae2c0d9-df61-4a95-a80e-d375a75f9722" containerName="payload-pre-processing"
Sep 18 20:27:35.023618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.023604    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd8789f-ad30-4250-9127-5992c719f1b1" containerName="maas-api"
Sep 18 20:27:35.023618 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.023610    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fc6684b-d2f4-4fc6-aa7a-11efb38d5d68" containerName="main"
Sep 18 20:27:35.026543 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.026521    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.030172 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.030149    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-de7ea3-serving-cert\""
Sep 18 20:27:35.038444 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.038419    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"]
Sep 18 20:27:35.059745 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.059720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-metrics-certs\") pod \"maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.059893 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.059751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.059963 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.059897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.059963 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.059927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpr57\" (UniqueName: \"kubernetes.io/projected/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-kube-api-access-vpr57\") pod \"maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.060553 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.060526    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:35.132744 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.132691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:35.160807 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.160774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-metrics-certs\") pod \"maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.160807 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.160809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.161087 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.160887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.161087 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.160911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpr57\" (UniqueName: \"kubernetes.io/projected/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-kube-api-access-vpr57\") pod \"maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.161087 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:27:35.160997    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-serving-cert: secret "maas-api-e2e-ipp-a-de7ea3-serving-cert" not found
Sep 18 20:27:35.161087 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:27:35.161071    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-maas-api-tls podName:7ed0fd99-1aba-4808-adb4-f1f58c1d928c nodeName:}" failed. No retries permitted until 2026-09-18 20:27:35.661054692 +0000 UTC m=+1857.499518622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-maas-api-tls") pod "maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q" (UID: "7ed0fd99-1aba-4808-adb4-f1f58c1d928c") : secret "maas-api-e2e-ipp-a-de7ea3-serving-cert" not found
Sep 18 20:27:35.161563 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.161546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.163331 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.163312    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-metrics-certs\") pod \"maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.169625 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.169601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpr57\" (UniqueName: \"kubernetes.io/projected/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-kube-api-access-vpr57\") pod \"maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.397102 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.397012    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:35.664891 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.664812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.667243 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.667219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.692458 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.692429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:35.789439 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.789396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:27:35.936873 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.936789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:35.974167 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.974135    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"]
Sep 18 20:27:35.977754 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.977657    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"
Sep 18 20:27:35.980307 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.980281    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-de7ea3\""
Sep 18 20:27:35.980481 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.980386    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-de7ea3-dockercfg-st6sw\""
Sep 18 20:27:35.988923 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:35.988894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"]
Sep 18 20:27:36.068425 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.068390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eddcc836-2b2c-4ad1-a887-c7d060800e0a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr\" (UID: \"eddcc836-2b2c-4ad1-a887-c7d060800e0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"
Sep 18 20:27:36.068787 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.068466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rpqg\" (UniqueName: \"kubernetes.io/projected/eddcc836-2b2c-4ad1-a887-c7d060800e0a-kube-api-access-8rpqg\") pod \"payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr\" (UID: \"eddcc836-2b2c-4ad1-a887-c7d060800e0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"
Sep 18 20:27:36.169805 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.169769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rpqg\" (UniqueName: \"kubernetes.io/projected/eddcc836-2b2c-4ad1-a887-c7d060800e0a-kube-api-access-8rpqg\") pod \"payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr\" (UID: \"eddcc836-2b2c-4ad1-a887-c7d060800e0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"
Sep 18 20:27:36.169995 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.169843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eddcc836-2b2c-4ad1-a887-c7d060800e0a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr\" (UID: \"eddcc836-2b2c-4ad1-a887-c7d060800e0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"
Sep 18 20:27:36.170608 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.170589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eddcc836-2b2c-4ad1-a887-c7d060800e0a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr\" (UID: \"eddcc836-2b2c-4ad1-a887-c7d060800e0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"
Sep 18 20:27:36.178214 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.178195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rpqg\" (UniqueName: \"kubernetes.io/projected/eddcc836-2b2c-4ad1-a887-c7d060800e0a-kube-api-access-8rpqg\") pod \"payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr\" (UID: \"eddcc836-2b2c-4ad1-a887-c7d060800e0a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"
Sep 18 20:27:36.221604 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.221535    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"]
Sep 18 20:27:36.223968 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:27:36.223941    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed0fd99_1aba_4808_adb4_f1f58c1d928c.slice/crio-4bbf3a72feac09d34bbb2cc0ec9e22877ab937ad89350084e09cf5cdf62a5235 WatchSource:0}: Error finding container 4bbf3a72feac09d34bbb2cc0ec9e22877ab937ad89350084e09cf5cdf62a5235: Status 404 returned error can't find the container with id 4bbf3a72feac09d34bbb2cc0ec9e22877ab937ad89350084e09cf5cdf62a5235
Sep 18 20:27:36.292258 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.292233    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"
Sep 18 20:27:36.570665 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.570638    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"]
Sep 18 20:27:36.572009 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:27:36.571984    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddcc836_2b2c_4ad1_a887_c7d060800e0a.slice/crio-a9a11a4d42e184844c31c73ca50ff09a90d0e8aa020248877e141eb9a5ad0563 WatchSource:0}: Error finding container a9a11a4d42e184844c31c73ca50ff09a90d0e8aa020248877e141eb9a5ad0563: Status 404 returned error can't find the container with id a9a11a4d42e184844c31c73ca50ff09a90d0e8aa020248877e141eb9a5ad0563
Sep 18 20:27:36.955883 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.955786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q" event={"ID":"7ed0fd99-1aba-4808-adb4-f1f58c1d928c","Type":"ContainerStarted","Data":"c8e12b4f1d7f631ab1fcbfa3d8cb9d2bf79922286867cf2c0f264c0b4d1f349c"}
Sep 18 20:27:36.955883 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.955850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q" event={"ID":"7ed0fd99-1aba-4808-adb4-f1f58c1d928c","Type":"ContainerStarted","Data":"4bbf3a72feac09d34bbb2cc0ec9e22877ab937ad89350084e09cf5cdf62a5235"}
Sep 18 20:27:36.956152 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.955966    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:36.957380 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.957358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr" event={"ID":"eddcc836-2b2c-4ad1-a887-c7d060800e0a","Type":"ContainerStarted","Data":"38af16a9b2834c0f9ee3abd6e8435e814af66cce5100fa3ba082c0ed37861195"}
Sep 18 20:27:36.957495 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.957396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr" event={"ID":"eddcc836-2b2c-4ad1-a887-c7d060800e0a","Type":"ContainerStarted","Data":"a9a11a4d42e184844c31c73ca50ff09a90d0e8aa020248877e141eb9a5ad0563"}
Sep 18 20:27:36.957495 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.957422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"
Sep 18 20:27:36.979144 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:36.979010    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q" podStartSLOduration=2.54366459 podStartE2EDuration="2.978993204s" totalImagesPullingTime="435.328614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:27:36.977866799 +0000 UTC m=+1858.816330764" watchObservedRunningTime="2026-09-18 20:27:36.978993204 +0000 UTC m=+1858.817457156"
Sep 18 20:27:37.000625 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:37.000571    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr" podStartSLOduration=2.000557246 podStartE2EDuration="2.000557246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:27:36.999913057 +0000 UTC m=+1858.838377012" watchObservedRunningTime="2026-09-18 20:27:37.000557246 +0000 UTC m=+1858.839021199"
Sep 18 20:27:42.967966 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:42.967920    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:27:47.964127 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:47.964094    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"
Sep 18 20:27:59.128712 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:27:59.128627    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:00.075339 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:00.075304    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:00.883358 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:00.883323    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:01.765603 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:01.765564    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:02.668859 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:02.668825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:03.346824 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:03.346793    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:03.853165 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:03.853133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:04.041162 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:04.041119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:06.729007 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:06.728968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:06.875128 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:06.875090    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:06.991976 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:06.991875    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:07.815236 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:07.815203    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:08.825407 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:08.825372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:10.118758 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:10.118729    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:10.218449 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:10.218414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:10.330413 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:10.330378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:10.809023 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:10.808981    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:10.974688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:10.974655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:11.077799 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:11.077728    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"]
Sep 18 20:28:11.081236 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:11.081220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"
Sep 18 20:28:11.083966 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:11.083923    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a1a0a2-dockercfg-8z6c7\""
Sep 18 20:28:11.084088 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:11.083976    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a1a0a2\""
Sep 18 20:28:11.095383 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:11.095358    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"]
Sep 18 20:28:11.175089 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:11.175053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4576a16-b734-4e58-b0ee-db28bc243509-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m\" (UID: \"d4576a16-b734-4e58-b0ee-db28bc243509\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"
Sep 18 20:28:11.175500 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:11.175104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnhkp\" (UniqueName: \"kubernetes.io/projected/d4576a16-b734-4e58-b0ee-db28bc243509-kube-api-access-nnhkp\") pod \"payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m\" (UID: \"d4576a16-b734-4e58-b0ee-db28bc243509\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"
Sep 18 20:28:11.276655 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:11.276613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4576a16-b734-4e58-b0ee-db28bc243509-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m\" (UID: \"d4576a16-b734-4e58-b0ee-db28bc243509\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"
Sep 18 20:28:11.276827 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:11.276666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnhkp\" (UniqueName: \"kubernetes.io/projected/d4576a16-b734-4e58-b0ee-db28bc243509-kube-api-access-nnhkp\") pod \"payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m\" (UID: \"d4576a16-b734-4e58-b0ee-db28bc243509\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"
Sep 18 20:28:11.277396 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:11.277373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4576a16-b734-4e58-b0ee-db28bc243509-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m\" (UID: \"d4576a16-b734-4e58-b0ee-db28bc243509\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"
Sep 18 20:28:11.285249 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:11.285224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnhkp\" (UniqueName: \"kubernetes.io/projected/d4576a16-b734-4e58-b0ee-db28bc243509-kube-api-access-nnhkp\") pod \"payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m\" (UID: \"d4576a16-b734-4e58-b0ee-db28bc243509\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"
Sep 18 20:28:11.391848 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:11.391761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"
Sep 18 20:28:11.667958 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:11.667914    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"]
Sep 18 20:28:11.669252 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:28:11.669224    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4576a16_b734_4e58_b0ee_db28bc243509.slice/crio-a2f3dca37156a7634e810820035c1c7a6f6243fb7ad4eb84dc9d8aea7781f7a2 WatchSource:0}: Error finding container a2f3dca37156a7634e810820035c1c7a6f6243fb7ad4eb84dc9d8aea7781f7a2: Status 404 returned error can't find the container with id a2f3dca37156a7634e810820035c1c7a6f6243fb7ad4eb84dc9d8aea7781f7a2
Sep 18 20:28:12.126517 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:12.126478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m" event={"ID":"d4576a16-b734-4e58-b0ee-db28bc243509","Type":"ContainerStarted","Data":"d9a90d2d71e9b99a530cefc4e3e2665968787a5a50c60d6b9663eb5469640351"}
Sep 18 20:28:12.126782 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:12.126538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m" event={"ID":"d4576a16-b734-4e58-b0ee-db28bc243509","Type":"ContainerStarted","Data":"a2f3dca37156a7634e810820035c1c7a6f6243fb7ad4eb84dc9d8aea7781f7a2"}
Sep 18 20:28:12.126782 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:12.126639    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"
Sep 18 20:28:12.146469 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:12.146413    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m" podStartSLOduration=1.146398122 podStartE2EDuration="1.146398122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:28:12.146015809 +0000 UTC m=+1893.984479762" watchObservedRunningTime="2026-09-18 20:28:12.146398122 +0000 UTC m=+1893.984862074"
Sep 18 20:28:16.680432 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:16.680392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:16.780525 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:16.780489    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:23.134032 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:23.134000    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"
Sep 18 20:28:33.558775 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:33.558736    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:35.037909 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:35.037867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:35.739292 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:35.739252    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:36.647585 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:36.647546    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:37.058158 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:37.058123    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:37.560213 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:37.560179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:44.323879 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:44.323844    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:46.551057 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:46.551013    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:28:51.952361 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:28:51.952321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:01.446576 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:01.446534    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:18.176230 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:18.176199    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr_eddcc836-2b2c-4ad1-a887-c7d060800e0a/payload-processing/0.log"
Sep 18 20:29:18.897539 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:18.897500    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:19.388133 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:19.388104    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:19.597178 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:19.597145    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:21.923653 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:21.923610    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:22.488483 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:22.488448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:22.618439 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:22.618402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:23.363335 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:23.363300    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:24.543142 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:24.543110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:25.772367 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:25.772334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:26.854803 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:26.854770    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:27.742081 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.742040    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"]
Sep 18 20:29:27.745539 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.745520    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:27.747923 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.747901    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-876382-serving-cert\""
Sep 18 20:29:27.761311 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.761289    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"]
Sep 18 20:29:27.783083 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.783048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:27.836761 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.836731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1f615d-a08a-4d0e-8901-e99b85015ffd-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:27.836976 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.836779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:27.836976 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.836807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:27.837094 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.836982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cksq7\" (UniqueName: \"kubernetes.io/projected/6c1f615d-a08a-4d0e-8901-e99b85015ffd-kube-api-access-cksq7\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:27.883438 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.883402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:27.937616 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.937586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cksq7\" (UniqueName: \"kubernetes.io/projected/6c1f615d-a08a-4d0e-8901-e99b85015ffd-kube-api-access-cksq7\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:27.937616 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.937619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1f615d-a08a-4d0e-8901-e99b85015ffd-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:27.937846 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.937652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:27.937846 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.937679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:27.937846 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:29:27.937760    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-serving-cert: secret "maas-api-e2e-ipp-cleanup-876382-serving-cert" not found
Sep 18 20:29:27.937846 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:29:27.937836    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-maas-api-tls podName:6c1f615d-a08a-4d0e-8901-e99b85015ffd nodeName:}" failed. No retries permitted until 2026-09-18 20:29:28.437820286 +0000 UTC m=+1970.276284215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h" (UID: "6c1f615d-a08a-4d0e-8901-e99b85015ffd") : secret "maas-api-e2e-ipp-cleanup-876382-serving-cert" not found
Sep 18 20:29:27.938284 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.938266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1f615d-a08a-4d0e-8901-e99b85015ffd-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:27.940045 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.940021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:27.950232 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:27.950209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cksq7\" (UniqueName: \"kubernetes.io/projected/6c1f615d-a08a-4d0e-8901-e99b85015ffd-kube-api-access-cksq7\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:28.323718 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.323670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:28.423174 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.423134    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:28.442343 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.442312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:28.442494 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:29:28.442470    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-serving-cert: secret "maas-api-e2e-ipp-cleanup-876382-serving-cert" not found
Sep 18 20:29:28.442550 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:29:28.442539    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-maas-api-tls podName:6c1f615d-a08a-4d0e-8901-e99b85015ffd nodeName:}" failed. No retries permitted until 2026-09-18 20:29:29.442521652 +0000 UTC m=+1971.280985599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h" (UID: "6c1f615d-a08a-4d0e-8901-e99b85015ffd") : secret "maas-api-e2e-ipp-cleanup-876382-serving-cert" not found
Sep 18 20:29:28.458057 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.458016    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"]
Sep 18 20:29:28.461790 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.461773    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"
Sep 18 20:29:28.464031 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.463998    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-876382\""
Sep 18 20:29:28.464143 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.464118    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-876382-dockercfg-s6f22\""
Sep 18 20:29:28.474335 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.474305    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"]
Sep 18 20:29:28.530269 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.530237    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:28.543492 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.543458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-775k2\" (UniqueName: \"kubernetes.io/projected/c2efb612-e0c1-4358-a4be-02e86a64cff7-kube-api-access-775k2\") pod \"payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx\" (UID: \"c2efb612-e0c1-4358-a4be-02e86a64cff7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"
Sep 18 20:29:28.543631 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.543526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2efb612-e0c1-4358-a4be-02e86a64cff7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx\" (UID: \"c2efb612-e0c1-4358-a4be-02e86a64cff7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"
Sep 18 20:29:28.644167 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.644082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2efb612-e0c1-4358-a4be-02e86a64cff7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx\" (UID: \"c2efb612-e0c1-4358-a4be-02e86a64cff7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"
Sep 18 20:29:28.644322 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.644195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-775k2\" (UniqueName: \"kubernetes.io/projected/c2efb612-e0c1-4358-a4be-02e86a64cff7-kube-api-access-775k2\") pod \"payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx\" (UID: \"c2efb612-e0c1-4358-a4be-02e86a64cff7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"
Sep 18 20:29:28.644782 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.644763    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2efb612-e0c1-4358-a4be-02e86a64cff7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx\" (UID: \"c2efb612-e0c1-4358-a4be-02e86a64cff7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"
Sep 18 20:29:28.653602 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.653573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-775k2\" (UniqueName: \"kubernetes.io/projected/c2efb612-e0c1-4358-a4be-02e86a64cff7-kube-api-access-775k2\") pod \"payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx\" (UID: \"c2efb612-e0c1-4358-a4be-02e86a64cff7\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"
Sep 18 20:29:28.773088 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:28.773048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"
Sep 18 20:29:29.052787 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:29.052760    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"]
Sep 18 20:29:29.055308 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:29:29.055282    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2efb612_e0c1_4358_a4be_02e86a64cff7.slice/crio-3477b2dde9a64e0ae1584cc14ff72ed9e7bd1515f3e20149b47876b40fef5fdf WatchSource:0}: Error finding container 3477b2dde9a64e0ae1584cc14ff72ed9e7bd1515f3e20149b47876b40fef5fdf: Status 404 returned error can't find the container with id 3477b2dde9a64e0ae1584cc14ff72ed9e7bd1515f3e20149b47876b40fef5fdf
Sep 18 20:29:29.451225 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:29.451130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:29.453619 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:29.453590    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:29.497394 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:29.497343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx" event={"ID":"c2efb612-e0c1-4358-a4be-02e86a64cff7","Type":"ContainerStarted","Data":"e374540af7231a3b05331895eecd87ed6278531d07061667cea7c34c9e3879db"}
Sep 18 20:29:29.497566 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:29.497413    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx" event={"ID":"c2efb612-e0c1-4358-a4be-02e86a64cff7","Type":"ContainerStarted","Data":"3477b2dde9a64e0ae1584cc14ff72ed9e7bd1515f3e20149b47876b40fef5fdf"}
Sep 18 20:29:29.497566 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:29.497435    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"
Sep 18 20:29:29.518923 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:29.518867    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx" podStartSLOduration=1.518851011 podStartE2EDuration="1.518851011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:29.516030699 +0000 UTC m=+1971.354494651" watchObservedRunningTime="2026-09-18 20:29:29.518851011 +0000 UTC m=+1971.357314962"
Sep 18 20:29:29.556216 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:29.556181    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:29.834392 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:29.834365    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"]
Sep 18 20:29:29.836502 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:29:29.836478    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1f615d_a08a_4d0e_8901_e99b85015ffd.slice/crio-d3130552c516dcec726b870bd105342a008fc649917fe5c948db20e757afac9b WatchSource:0}: Error finding container d3130552c516dcec726b870bd105342a008fc649917fe5c948db20e757afac9b: Status 404 returned error can't find the container with id d3130552c516dcec726b870bd105342a008fc649917fe5c948db20e757afac9b
Sep 18 20:29:29.837896 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:29.837879    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:29:30.509646 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:30.509550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h" event={"ID":"6c1f615d-a08a-4d0e-8901-e99b85015ffd","Type":"ContainerStarted","Data":"35eff040e9d71fbf60f0c1fdefe38474dc98e657b6ceef454e9597d0e063cafa"}
Sep 18 20:29:30.509646 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:30.509630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h" event={"ID":"6c1f615d-a08a-4d0e-8901-e99b85015ffd","Type":"ContainerStarted","Data":"d3130552c516dcec726b870bd105342a008fc649917fe5c948db20e757afac9b"}
Sep 18 20:29:30.510083 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:30.509652    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:30.533312 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:30.533259    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h" podStartSLOduration=3.142677794 podStartE2EDuration="3.533245587s" totalImagesPullingTime="390.567793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:30.530588786 +0000 UTC m=+1972.369052737" watchObservedRunningTime="2026-09-18 20:29:30.533245587 +0000 UTC m=+1972.371709538"
Sep 18 20:29:36.519822 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:36.519787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:29:40.512568 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:40.512534    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"
Sep 18 20:29:44.169491 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:44.169448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:49.500921 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:49.500869    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:51.478552 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:51.478487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:52.736682 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:52.736642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:53.056134 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:53.056041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:54.842041 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:54.842004    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:56.098394 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:56.098352    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:57.095075 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:57.095038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:58.129389 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:58.129356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:29:58.972401 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:29:58.972365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:00.009368 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:00.009337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:00.290919 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:00.290842    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a1a0a2-29829390-kdfb8"]
Sep 18 20:30:00.295270 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:00.295241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a1a0a2-29829390-kdfb8"
Sep 18 20:30:00.297425 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:00.297400    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:30:00.304756 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:00.304734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a1a0a2-29829390-kdfb8"
Sep 18 20:30:00.322207 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:00.322165    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a1a0a2-29829390-kdfb8"]
Sep 18 20:30:00.599224 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:00.599196    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a1a0a2-29829390-kdfb8"]
Sep 18 20:30:00.601005 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:30:00.600976    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3411e0c_5d64_4293_9b44_0003b4d0d046.slice/crio-85726d066bc71fe57eb084112b30829fcd6358fffa446b8b51799b4b86a4bdcb WatchSource:0}: Error finding container 85726d066bc71fe57eb084112b30829fcd6358fffa446b8b51799b4b86a4bdcb: Status 404 returned error can't find the container with id 85726d066bc71fe57eb084112b30829fcd6358fffa446b8b51799b4b86a4bdcb
Sep 18 20:30:00.656406 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:00.656376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a1a0a2-29829390-kdfb8" event={"ID":"d3411e0c-5d64-4293-9b44-0003b4d0d046","Type":"ContainerStarted","Data":"85726d066bc71fe57eb084112b30829fcd6358fffa446b8b51799b4b86a4bdcb"}
Sep 18 20:30:01.663277 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:01.663246    2578 generic.go:356] "Generic (PLEG): container finished" podID="d3411e0c-5d64-4293-9b44-0003b4d0d046" containerID="313359d340b8ce53ab95b642812d77974b5157b62b47fd4a417478ee68167ca6" exitCode=0
Sep 18 20:30:01.663669 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:01.663337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a1a0a2-29829390-kdfb8" event={"ID":"d3411e0c-5d64-4293-9b44-0003b4d0d046","Type":"ContainerDied","Data":"313359d340b8ce53ab95b642812d77974b5157b62b47fd4a417478ee68167ca6"}
Sep 18 20:30:02.810573 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:02.810550    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a1a0a2-29829390-kdfb8"
Sep 18 20:30:03.675827 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:03.675799    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a1a0a2-29829390-kdfb8"
Sep 18 20:30:03.676015 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:03.675798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a1a0a2-29829390-kdfb8" event={"ID":"d3411e0c-5d64-4293-9b44-0003b4d0d046","Type":"ContainerDied","Data":"85726d066bc71fe57eb084112b30829fcd6358fffa446b8b51799b4b86a4bdcb"}
Sep 18 20:30:03.676015 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:03.675925    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85726d066bc71fe57eb084112b30829fcd6358fffa446b8b51799b4b86a4bdcb"
Sep 18 20:30:03.909148 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:03.909113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:04.691398 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:04.691363    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"]
Sep 18 20:30:04.692259 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:04.691828    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h" podUID="6c1f615d-a08a-4d0e-8901-e99b85015ffd" containerName="maas-api" containerID="cri-o://35eff040e9d71fbf60f0c1fdefe38474dc98e657b6ceef454e9597d0e063cafa" gracePeriod=30
Sep 18 20:30:04.738845 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:04.738814    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:04.793657 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:04.793622    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"]
Sep 18 20:30:04.793913 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:04.793887    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx" podUID="c2efb612-e0c1-4358-a4be-02e86a64cff7" containerName="payload-processing" containerID="cri-o://e374540af7231a3b05331895eecd87ed6278531d07061667cea7c34c9e3879db" gracePeriod=30
Sep 18 20:30:04.870324 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:04.870230    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:04.954802 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:04.954745    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:30:05.001567 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.001533    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:05.032363 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.032339    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"
Sep 18 20:30:05.064392 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.064358    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cksq7\" (UniqueName: \"kubernetes.io/projected/6c1f615d-a08a-4d0e-8901-e99b85015ffd-kube-api-access-cksq7\") pod \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") "
Sep 18 20:30:05.064579 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.064443    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-maas-api-tls\") pod \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") "
Sep 18 20:30:05.064579 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.064503    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1f615d-a08a-4d0e-8901-e99b85015ffd-openshift-service-ca\") pod \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") "
Sep 18 20:30:05.064579 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.064547    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-metrics-certs\") pod \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\" (UID: \"6c1f615d-a08a-4d0e-8901-e99b85015ffd\") "
Sep 18 20:30:05.064950 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.064898    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1f615d-a08a-4d0e-8901-e99b85015ffd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c1f615d-a08a-4d0e-8901-e99b85015ffd" (UID: "6c1f615d-a08a-4d0e-8901-e99b85015ffd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:30:05.066868 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.066835    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c1f615d-a08a-4d0e-8901-e99b85015ffd" (UID: "6c1f615d-a08a-4d0e-8901-e99b85015ffd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:30:05.066995 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.066870    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6c1f615d-a08a-4d0e-8901-e99b85015ffd" (UID: "6c1f615d-a08a-4d0e-8901-e99b85015ffd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:30:05.066995 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.066966    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1f615d-a08a-4d0e-8901-e99b85015ffd-kube-api-access-cksq7" (OuterVolumeSpecName: "kube-api-access-cksq7") pod "6c1f615d-a08a-4d0e-8901-e99b85015ffd" (UID: "6c1f615d-a08a-4d0e-8901-e99b85015ffd"). InnerVolumeSpecName "kube-api-access-cksq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:30:05.110469 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.110428    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:05.165718 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.165685    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-775k2\" (UniqueName: \"kubernetes.io/projected/c2efb612-e0c1-4358-a4be-02e86a64cff7-kube-api-access-775k2\") pod \"c2efb612-e0c1-4358-a4be-02e86a64cff7\" (UID: \"c2efb612-e0c1-4358-a4be-02e86a64cff7\") "
Sep 18 20:30:05.165885 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.165756    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2efb612-e0c1-4358-a4be-02e86a64cff7-plugins-config-volume\") pod \"c2efb612-e0c1-4358-a4be-02e86a64cff7\" (UID: \"c2efb612-e0c1-4358-a4be-02e86a64cff7\") "
Sep 18 20:30:05.166001 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.165984    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cksq7\" (UniqueName: \"kubernetes.io/projected/6c1f615d-a08a-4d0e-8901-e99b85015ffd-kube-api-access-cksq7\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:30:05.166068 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.166007    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:30:05.166068 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.166020    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1f615d-a08a-4d0e-8901-e99b85015ffd-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:30:05.166068 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.166028    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1f615d-a08a-4d0e-8901-e99b85015ffd-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:30:05.166218 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.166132    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2efb612-e0c1-4358-a4be-02e86a64cff7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2efb612-e0c1-4358-a4be-02e86a64cff7" (UID: "c2efb612-e0c1-4358-a4be-02e86a64cff7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:30:05.167972 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.167949    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2efb612-e0c1-4358-a4be-02e86a64cff7-kube-api-access-775k2" (OuterVolumeSpecName: "kube-api-access-775k2") pod "c2efb612-e0c1-4358-a4be-02e86a64cff7" (UID: "c2efb612-e0c1-4358-a4be-02e86a64cff7"). InnerVolumeSpecName "kube-api-access-775k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:30:05.208542 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.208448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:05.267234 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.267187    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2efb612-e0c1-4358-a4be-02e86a64cff7-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:30:05.267234 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.267222    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-775k2\" (UniqueName: \"kubernetes.io/projected/c2efb612-e0c1-4358-a4be-02e86a64cff7-kube-api-access-775k2\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:30:05.483909 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.483877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:05.687586 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.687553    2578 generic.go:356] "Generic (PLEG): container finished" podID="c2efb612-e0c1-4358-a4be-02e86a64cff7" containerID="e374540af7231a3b05331895eecd87ed6278531d07061667cea7c34c9e3879db" exitCode=0
Sep 18 20:30:05.687747 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.687599    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"
Sep 18 20:30:05.687747 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.687639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx" event={"ID":"c2efb612-e0c1-4358-a4be-02e86a64cff7","Type":"ContainerDied","Data":"e374540af7231a3b05331895eecd87ed6278531d07061667cea7c34c9e3879db"}
Sep 18 20:30:05.687747 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.687694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx" event={"ID":"c2efb612-e0c1-4358-a4be-02e86a64cff7","Type":"ContainerDied","Data":"3477b2dde9a64e0ae1584cc14ff72ed9e7bd1515f3e20149b47876b40fef5fdf"}
Sep 18 20:30:05.687747 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.687724    2578 scope.go:117] "RemoveContainer" containerID="e374540af7231a3b05331895eecd87ed6278531d07061667cea7c34c9e3879db"
Sep 18 20:30:05.689066 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.689040    2578 generic.go:356] "Generic (PLEG): container finished" podID="6c1f615d-a08a-4d0e-8901-e99b85015ffd" containerID="35eff040e9d71fbf60f0c1fdefe38474dc98e657b6ceef454e9597d0e063cafa" exitCode=0
Sep 18 20:30:05.689191 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.689093    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"
Sep 18 20:30:05.689191 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.689077    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h" event={"ID":"6c1f615d-a08a-4d0e-8901-e99b85015ffd","Type":"ContainerDied","Data":"35eff040e9d71fbf60f0c1fdefe38474dc98e657b6ceef454e9597d0e063cafa"}
Sep 18 20:30:05.689191 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.689149    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h" event={"ID":"6c1f615d-a08a-4d0e-8901-e99b85015ffd","Type":"ContainerDied","Data":"d3130552c516dcec726b870bd105342a008fc649917fe5c948db20e757afac9b"}
Sep 18 20:30:05.698374 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.698240    2578 scope.go:117] "RemoveContainer" containerID="e374540af7231a3b05331895eecd87ed6278531d07061667cea7c34c9e3879db"
Sep 18 20:30:05.698565 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:30:05.698548    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e374540af7231a3b05331895eecd87ed6278531d07061667cea7c34c9e3879db\": container with ID starting with e374540af7231a3b05331895eecd87ed6278531d07061667cea7c34c9e3879db not found: ID does not exist" containerID="e374540af7231a3b05331895eecd87ed6278531d07061667cea7c34c9e3879db"
Sep 18 20:30:05.698606 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.698573    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e374540af7231a3b05331895eecd87ed6278531d07061667cea7c34c9e3879db"} err="failed to get container status \"e374540af7231a3b05331895eecd87ed6278531d07061667cea7c34c9e3879db\": rpc error: code = NotFound desc = could not find container \"e374540af7231a3b05331895eecd87ed6278531d07061667cea7c34c9e3879db\": container with ID starting with e374540af7231a3b05331895eecd87ed6278531d07061667cea7c34c9e3879db not found: ID does not exist"
Sep 18 20:30:05.698606 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.698589    2578 scope.go:117] "RemoveContainer" containerID="35eff040e9d71fbf60f0c1fdefe38474dc98e657b6ceef454e9597d0e063cafa"
Sep 18 20:30:05.706852 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.706834    2578 scope.go:117] "RemoveContainer" containerID="35eff040e9d71fbf60f0c1fdefe38474dc98e657b6ceef454e9597d0e063cafa"
Sep 18 20:30:05.707199 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:30:05.707165    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35eff040e9d71fbf60f0c1fdefe38474dc98e657b6ceef454e9597d0e063cafa\": container with ID starting with 35eff040e9d71fbf60f0c1fdefe38474dc98e657b6ceef454e9597d0e063cafa not found: ID does not exist" containerID="35eff040e9d71fbf60f0c1fdefe38474dc98e657b6ceef454e9597d0e063cafa"
Sep 18 20:30:05.707299 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.707199    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35eff040e9d71fbf60f0c1fdefe38474dc98e657b6ceef454e9597d0e063cafa"} err="failed to get container status \"35eff040e9d71fbf60f0c1fdefe38474dc98e657b6ceef454e9597d0e063cafa\": rpc error: code = NotFound desc = could not find container \"35eff040e9d71fbf60f0c1fdefe38474dc98e657b6ceef454e9597d0e063cafa\": container with ID starting with 35eff040e9d71fbf60f0c1fdefe38474dc98e657b6ceef454e9597d0e063cafa not found: ID does not exist"
Sep 18 20:30:05.714985 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.714956    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"]
Sep 18 20:30:05.718478 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.718455    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-876382-7797846bd7-h7hgx"]
Sep 18 20:30:05.720064 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.720032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:05.736621 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.736529    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"]
Sep 18 20:30:05.743024 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:05.742982    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-876382-5f487c7969-twp5h"]
Sep 18 20:30:06.318410 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:06.318380    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c1f615d-a08a-4d0e-8901-e99b85015ffd" path="/var/lib/kubelet/pods/6c1f615d-a08a-4d0e-8901-e99b85015ffd/volumes"
Sep 18 20:30:06.318774 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:06.318739    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2efb612-e0c1-4358-a4be-02e86a64cff7" path="/var/lib/kubelet/pods/c2efb612-e0c1-4358-a4be-02e86a64cff7/volumes"
Sep 18 20:30:06.591433 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:06.591349    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:07.661471 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:07.661440    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:08.543390 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:08.543354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:09.542369 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:09.542330    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:12.042673 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:12.042625    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:12.117150 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:12.117097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:12.338069 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:12.337974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:16.061257 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:16.061212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:16.954078 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:16.954031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:18.062296 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:18.062240    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:18.861653 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:18.861615    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:19.943641 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:19.943604    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:20.722483 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:20.722446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:31.635378 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:31.635342    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"]
Sep 18 20:30:31.635851 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:31.635645    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q" podUID="7ed0fd99-1aba-4808-adb4-f1f58c1d928c" containerName="maas-api" containerID="cri-o://c8e12b4f1d7f631ab1fcbfa3d8cb9d2bf79922286867cf2c0f264c0b4d1f349c" gracePeriod=30
Sep 18 20:30:31.681425 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:31.681386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:31.708171 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:31.708137    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"]
Sep 18 20:30:31.708378 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:31.708355    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr" podUID="eddcc836-2b2c-4ad1-a887-c7d060800e0a" containerName="payload-processing" containerID="cri-o://38af16a9b2834c0f9ee3abd6e8435e814af66cce5100fa3ba082c0ed37861195" gracePeriod=30
Sep 18 20:30:31.792303 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:31.792271    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:31.825988 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:31.825955    2578 generic.go:356] "Generic (PLEG): container finished" podID="eddcc836-2b2c-4ad1-a887-c7d060800e0a" containerID="38af16a9b2834c0f9ee3abd6e8435e814af66cce5100fa3ba082c0ed37861195" exitCode=0
Sep 18 20:30:31.825988 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:31.825979    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr" event={"ID":"eddcc836-2b2c-4ad1-a887-c7d060800e0a","Type":"ContainerDied","Data":"38af16a9b2834c0f9ee3abd6e8435e814af66cce5100fa3ba082c0ed37861195"}
Sep 18 20:30:31.827880 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:31.827858    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ed0fd99-1aba-4808-adb4-f1f58c1d928c" containerID="c8e12b4f1d7f631ab1fcbfa3d8cb9d2bf79922286867cf2c0f264c0b4d1f349c" exitCode=0
Sep 18 20:30:31.828023 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:31.827974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q" event={"ID":"7ed0fd99-1aba-4808-adb4-f1f58c1d928c","Type":"ContainerDied","Data":"c8e12b4f1d7f631ab1fcbfa3d8cb9d2bf79922286867cf2c0f264c0b4d1f349c"}
Sep 18 20:30:31.922438 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:31.922391    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:31.963191 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:31.963162    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:30:31.990465 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:31.990442    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"
Sep 18 20:30:32.003494 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.003466    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-metrics-certs\") pod \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") "
Sep 18 20:30:32.003620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.003545    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpr57\" (UniqueName: \"kubernetes.io/projected/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-kube-api-access-vpr57\") pod \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") "
Sep 18 20:30:32.003620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.003610    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-maas-api-tls\") pod \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") "
Sep 18 20:30:32.003745 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.003662    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-openshift-service-ca\") pod \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\" (UID: \"7ed0fd99-1aba-4808-adb4-f1f58c1d928c\") "
Sep 18 20:30:32.004394 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.004233    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ed0fd99-1aba-4808-adb4-f1f58c1d928c" (UID: "7ed0fd99-1aba-4808-adb4-f1f58c1d928c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:30:32.015861 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.015833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ed0fd99-1aba-4808-adb4-f1f58c1d928c" (UID: "7ed0fd99-1aba-4808-adb4-f1f58c1d928c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:30:32.015981 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.015854    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ed0fd99-1aba-4808-adb4-f1f58c1d928c" (UID: "7ed0fd99-1aba-4808-adb4-f1f58c1d928c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:30:32.015981 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.015833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-kube-api-access-vpr57" (OuterVolumeSpecName: "kube-api-access-vpr57") pod "7ed0fd99-1aba-4808-adb4-f1f58c1d928c" (UID: "7ed0fd99-1aba-4808-adb4-f1f58c1d928c"). InnerVolumeSpecName "kube-api-access-vpr57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:30:32.052370 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.052334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:32.104155 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.104120    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rpqg\" (UniqueName: \"kubernetes.io/projected/eddcc836-2b2c-4ad1-a887-c7d060800e0a-kube-api-access-8rpqg\") pod \"eddcc836-2b2c-4ad1-a887-c7d060800e0a\" (UID: \"eddcc836-2b2c-4ad1-a887-c7d060800e0a\") "
Sep 18 20:30:32.104320 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.104203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eddcc836-2b2c-4ad1-a887-c7d060800e0a-plugins-config-volume\") pod \"eddcc836-2b2c-4ad1-a887-c7d060800e0a\" (UID: \"eddcc836-2b2c-4ad1-a887-c7d060800e0a\") "
Sep 18 20:30:32.104406 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.104391    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:30:32.104469 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.104410    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:30:32.104469 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.104424    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:30:32.104469 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.104437    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpr57\" (UniqueName: \"kubernetes.io/projected/7ed0fd99-1aba-4808-adb4-f1f58c1d928c-kube-api-access-vpr57\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:30:32.104612 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.104550    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eddcc836-2b2c-4ad1-a887-c7d060800e0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eddcc836-2b2c-4ad1-a887-c7d060800e0a" (UID: "eddcc836-2b2c-4ad1-a887-c7d060800e0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:30:32.106282 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.106260    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eddcc836-2b2c-4ad1-a887-c7d060800e0a-kube-api-access-8rpqg" (OuterVolumeSpecName: "kube-api-access-8rpqg") pod "eddcc836-2b2c-4ad1-a887-c7d060800e0a" (UID: "eddcc836-2b2c-4ad1-a887-c7d060800e0a"). InnerVolumeSpecName "kube-api-access-8rpqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:30:32.205820 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.205734    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rpqg\" (UniqueName: \"kubernetes.io/projected/eddcc836-2b2c-4ad1-a887-c7d060800e0a-kube-api-access-8rpqg\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:30:32.205820 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.205765    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eddcc836-2b2c-4ad1-a887-c7d060800e0a-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:30:32.405809 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.405776    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:32.481011 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.480964    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:32.834586 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.834498    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"
Sep 18 20:30:32.834586 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.834532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q" event={"ID":"7ed0fd99-1aba-4808-adb4-f1f58c1d928c","Type":"ContainerDied","Data":"4bbf3a72feac09d34bbb2cc0ec9e22877ab937ad89350084e09cf5cdf62a5235"}
Sep 18 20:30:32.835085 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.834593    2578 scope.go:117] "RemoveContainer" containerID="c8e12b4f1d7f631ab1fcbfa3d8cb9d2bf79922286867cf2c0f264c0b4d1f349c"
Sep 18 20:30:32.836093 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.836070    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr" event={"ID":"eddcc836-2b2c-4ad1-a887-c7d060800e0a","Type":"ContainerDied","Data":"a9a11a4d42e184844c31c73ca50ff09a90d0e8aa020248877e141eb9a5ad0563"}
Sep 18 20:30:32.836163 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.836100    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"
Sep 18 20:30:32.843803 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.843787    2578 scope.go:117] "RemoveContainer" containerID="38af16a9b2834c0f9ee3abd6e8435e814af66cce5100fa3ba082c0ed37861195"
Sep 18 20:30:32.855657 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.855636    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"]
Sep 18 20:30:32.861104 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.861083    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-de7ea3-fcbdf454c-lncdr"]
Sep 18 20:30:32.874636 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.874614    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"]
Sep 18 20:30:32.880339 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:32.880318    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de7ea3-fcfb9bd7f-qw99q"]
Sep 18 20:30:33.421988 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:33.421954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:34.318444 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:34.318411    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ed0fd99-1aba-4808-adb4-f1f58c1d928c" path="/var/lib/kubelet/pods/7ed0fd99-1aba-4808-adb4-f1f58c1d928c/volumes"
Sep 18 20:30:34.318818 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:34.318773    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eddcc836-2b2c-4ad1-a887-c7d060800e0a" path="/var/lib/kubelet/pods/eddcc836-2b2c-4ad1-a887-c7d060800e0a/volumes"
Sep 18 20:30:34.597820 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:34.597706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:35.591681 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:35.591642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:30:37.427335 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:30:37.427279    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:14.363574 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:14.363485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:14.860112 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:14.860076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:21.215351 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:21.215295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:21.325369 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:21.325332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:21.657360 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:21.657261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:22.178091 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:22.178054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:23.346368 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:23.346329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:24.397405 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:24.397366    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:25.401227 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:25.401188    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:26.163892 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:26.163859    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:28.958542 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:28.958506    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:32.612814 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:32.612778    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:32.621399 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:32.621372    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a1a0a2-29829390-kdfb8"]
Sep 18 20:31:32.624717 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:32.624693    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a1a0a2-29829390-kdfb8"]
Sep 18 20:31:32.690191 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:32.690159    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"]
Sep 18 20:31:32.690426 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:32.690400    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m" podUID="d4576a16-b734-4e58-b0ee-db28bc243509" containerName="payload-pre-processing" containerID="cri-o://d9a90d2d71e9b99a530cefc4e3e2665968787a5a50c60d6b9663eb5469640351" gracePeriod=30
Sep 18 20:31:32.742331 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:32.742295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:32.837799 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:32.837763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:32.935248 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:32.935055    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:32.954840 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:32.954817    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"
Sep 18 20:31:32.994546 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:32.994517    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4576a16-b734-4e58-b0ee-db28bc243509-plugins-config-volume\") pod \"d4576a16-b734-4e58-b0ee-db28bc243509\" (UID: \"d4576a16-b734-4e58-b0ee-db28bc243509\") "
Sep 18 20:31:32.994546 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:32.994548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnhkp\" (UniqueName: \"kubernetes.io/projected/d4576a16-b734-4e58-b0ee-db28bc243509-kube-api-access-nnhkp\") pod \"d4576a16-b734-4e58-b0ee-db28bc243509\" (UID: \"d4576a16-b734-4e58-b0ee-db28bc243509\") "
Sep 18 20:31:32.995046 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:32.994988    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4576a16-b734-4e58-b0ee-db28bc243509-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4576a16-b734-4e58-b0ee-db28bc243509" (UID: "d4576a16-b734-4e58-b0ee-db28bc243509"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:32.996759 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:32.996739    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4576a16-b734-4e58-b0ee-db28bc243509-kube-api-access-nnhkp" (OuterVolumeSpecName: "kube-api-access-nnhkp") pod "d4576a16-b734-4e58-b0ee-db28bc243509" (UID: "d4576a16-b734-4e58-b0ee-db28bc243509"). InnerVolumeSpecName "kube-api-access-nnhkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:33.095239 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:33.095201    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4576a16-b734-4e58-b0ee-db28bc243509-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:31:33.095239 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:33.095235    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnhkp\" (UniqueName: \"kubernetes.io/projected/d4576a16-b734-4e58-b0ee-db28bc243509-kube-api-access-nnhkp\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:31:33.125621 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:33.125586    2578 generic.go:356] "Generic (PLEG): container finished" podID="d4576a16-b734-4e58-b0ee-db28bc243509" containerID="d9a90d2d71e9b99a530cefc4e3e2665968787a5a50c60d6b9663eb5469640351" exitCode=0
Sep 18 20:31:33.125807 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:33.125634    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"
Sep 18 20:31:33.125807 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:33.125674    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m" event={"ID":"d4576a16-b734-4e58-b0ee-db28bc243509","Type":"ContainerDied","Data":"d9a90d2d71e9b99a530cefc4e3e2665968787a5a50c60d6b9663eb5469640351"}
Sep 18 20:31:33.125807 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:33.125737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m" event={"ID":"d4576a16-b734-4e58-b0ee-db28bc243509","Type":"ContainerDied","Data":"a2f3dca37156a7634e810820035c1c7a6f6243fb7ad4eb84dc9d8aea7781f7a2"}
Sep 18 20:31:33.125807 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:33.125768    2578 scope.go:117] "RemoveContainer" containerID="d9a90d2d71e9b99a530cefc4e3e2665968787a5a50c60d6b9663eb5469640351"
Sep 18 20:31:33.134706 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:33.134684    2578 scope.go:117] "RemoveContainer" containerID="d9a90d2d71e9b99a530cefc4e3e2665968787a5a50c60d6b9663eb5469640351"
Sep 18 20:31:33.134989 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:31:33.134970    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a90d2d71e9b99a530cefc4e3e2665968787a5a50c60d6b9663eb5469640351\": container with ID starting with d9a90d2d71e9b99a530cefc4e3e2665968787a5a50c60d6b9663eb5469640351 not found: ID does not exist" containerID="d9a90d2d71e9b99a530cefc4e3e2665968787a5a50c60d6b9663eb5469640351"
Sep 18 20:31:33.135045 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:33.134999    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a90d2d71e9b99a530cefc4e3e2665968787a5a50c60d6b9663eb5469640351"} err="failed to get container status \"d9a90d2d71e9b99a530cefc4e3e2665968787a5a50c60d6b9663eb5469640351\": rpc error: code = NotFound desc = could not find container \"d9a90d2d71e9b99a530cefc4e3e2665968787a5a50c60d6b9663eb5469640351\": container with ID starting with d9a90d2d71e9b99a530cefc4e3e2665968787a5a50c60d6b9663eb5469640351 not found: ID does not exist"
Sep 18 20:31:33.151762 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:33.151739    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"]
Sep 18 20:31:33.157414 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:33.157391    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a1a0a2-6d8cdc97fd-jbb8m"]
Sep 18 20:31:33.324887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:33.324855    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:33.426306 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:33.426270    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:34.323824 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:34.323793    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3411e0c-5d64-4293-9b44-0003b4d0d046" path="/var/lib/kubelet/pods/d3411e0c-5d64-4293-9b44-0003b4d0d046/volumes"
Sep 18 20:31:34.324218 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:34.324134    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4576a16-b734-4e58-b0ee-db28bc243509" path="/var/lib/kubelet/pods/d4576a16-b734-4e58-b0ee-db28bc243509/volumes"
Sep 18 20:31:34.560656 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:34.560624    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:35.751194 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:35.751154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:40.283781 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:40.283745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:40.657333 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:40.657270    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:40.749230 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:40.749196    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:41.169238 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:41.169198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:42.246687 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:42.246657    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:43.200121 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:43.200085    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:44.199346 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:44.199311    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:51.721108 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:51.721073    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"]
Sep 18 20:31:51.739727 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:51.739693    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:51.827237 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:51.827204    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"]
Sep 18 20:31:51.827435 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:51.827409    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc" podUID="440a5a06-5f73-40f5-b7d3-90b9f6ae925a" containerName="payload-processing" containerID="cri-o://2b20707e3ea13f570cc34fbfa5009739f6847e4a228099be388d3556d6596f4c" gracePeriod=30
Sep 18 20:31:51.835634 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:51.835608    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"]
Sep 18 20:31:51.835819 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:51.835799    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn" podUID="62918655-ac3d-4e65-b7d1-84a1b797f5a9" containerName="payload-pre-processing" containerID="cri-o://49ae3dc706943e390d9805fe4150fc4e354d90c8c28f52d068f3e9c672c87b7d" gracePeriod=30
Sep 18 20:31:51.930033 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:51.929995    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:52.035113 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.035015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:52.106038 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.106014    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"
Sep 18 20:31:52.108681 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.108659    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"
Sep 18 20:31:52.157395 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.157366    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbhfn\" (UniqueName: \"kubernetes.io/projected/62918655-ac3d-4e65-b7d1-84a1b797f5a9-kube-api-access-xbhfn\") pod \"62918655-ac3d-4e65-b7d1-84a1b797f5a9\" (UID: \"62918655-ac3d-4e65-b7d1-84a1b797f5a9\") "
Sep 18 20:31:52.157556 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.157467    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62918655-ac3d-4e65-b7d1-84a1b797f5a9-plugins-config-volume\") pod \"62918655-ac3d-4e65-b7d1-84a1b797f5a9\" (UID: \"62918655-ac3d-4e65-b7d1-84a1b797f5a9\") "
Sep 18 20:31:52.157556 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.157492    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzznh\" (UniqueName: \"kubernetes.io/projected/440a5a06-5f73-40f5-b7d3-90b9f6ae925a-kube-api-access-vzznh\") pod \"440a5a06-5f73-40f5-b7d3-90b9f6ae925a\" (UID: \"440a5a06-5f73-40f5-b7d3-90b9f6ae925a\") "
Sep 18 20:31:52.157556 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.157511    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440a5a06-5f73-40f5-b7d3-90b9f6ae925a-plugins-config-volume\") pod \"440a5a06-5f73-40f5-b7d3-90b9f6ae925a\" (UID: \"440a5a06-5f73-40f5-b7d3-90b9f6ae925a\") "
Sep 18 20:31:52.157920 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.157890    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62918655-ac3d-4e65-b7d1-84a1b797f5a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62918655-ac3d-4e65-b7d1-84a1b797f5a9" (UID: "62918655-ac3d-4e65-b7d1-84a1b797f5a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:52.158073 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.157953    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440a5a06-5f73-40f5-b7d3-90b9f6ae925a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "440a5a06-5f73-40f5-b7d3-90b9f6ae925a" (UID: "440a5a06-5f73-40f5-b7d3-90b9f6ae925a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:52.159616 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.159594    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62918655-ac3d-4e65-b7d1-84a1b797f5a9-kube-api-access-xbhfn" (OuterVolumeSpecName: "kube-api-access-xbhfn") pod "62918655-ac3d-4e65-b7d1-84a1b797f5a9" (UID: "62918655-ac3d-4e65-b7d1-84a1b797f5a9"). InnerVolumeSpecName "kube-api-access-xbhfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:52.159616 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.159613    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440a5a06-5f73-40f5-b7d3-90b9f6ae925a-kube-api-access-vzznh" (OuterVolumeSpecName: "kube-api-access-vzznh") pod "440a5a06-5f73-40f5-b7d3-90b9f6ae925a" (UID: "440a5a06-5f73-40f5-b7d3-90b9f6ae925a"). InnerVolumeSpecName "kube-api-access-vzznh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:52.185613 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.185568    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:52.215625 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.215595    2578 generic.go:356] "Generic (PLEG): container finished" podID="62918655-ac3d-4e65-b7d1-84a1b797f5a9" containerID="49ae3dc706943e390d9805fe4150fc4e354d90c8c28f52d068f3e9c672c87b7d" exitCode=0
Sep 18 20:31:52.215872 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.215637    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"
Sep 18 20:31:52.215872 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.215636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn" event={"ID":"62918655-ac3d-4e65-b7d1-84a1b797f5a9","Type":"ContainerDied","Data":"49ae3dc706943e390d9805fe4150fc4e354d90c8c28f52d068f3e9c672c87b7d"}
Sep 18 20:31:52.215872 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.215779    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn" event={"ID":"62918655-ac3d-4e65-b7d1-84a1b797f5a9","Type":"ContainerDied","Data":"9a7615539b31b9d91a23e2b6353157c7be94ee8a277ace8d39de6d35ca93c27c"}
Sep 18 20:31:52.215872 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.215820    2578 scope.go:117] "RemoveContainer" containerID="49ae3dc706943e390d9805fe4150fc4e354d90c8c28f52d068f3e9c672c87b7d"
Sep 18 20:31:52.217240 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.217075    2578 generic.go:356] "Generic (PLEG): container finished" podID="440a5a06-5f73-40f5-b7d3-90b9f6ae925a" containerID="2b20707e3ea13f570cc34fbfa5009739f6847e4a228099be388d3556d6596f4c" exitCode=0
Sep 18 20:31:52.217240 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.217144    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"
Sep 18 20:31:52.217240 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.217176    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc" event={"ID":"440a5a06-5f73-40f5-b7d3-90b9f6ae925a","Type":"ContainerDied","Data":"2b20707e3ea13f570cc34fbfa5009739f6847e4a228099be388d3556d6596f4c"}
Sep 18 20:31:52.217418 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.217243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc" event={"ID":"440a5a06-5f73-40f5-b7d3-90b9f6ae925a","Type":"ContainerDied","Data":"0bdaca510af7dfa411fb33102c1ab2677d08e6cb4cc1b8c21d05ae3a0af269e7"}
Sep 18 20:31:52.217476 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.217427    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch" podUID="a12d929b-4cdd-4aeb-bc3e-b23dcce48b77" containerName="maas-api" containerID="cri-o://2ed227f3a53c507daa0aec96011a5bdc82f95f97ab9e553f844ff062e7d573b2" gracePeriod=30
Sep 18 20:31:52.226099 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.226079    2578 scope.go:117] "RemoveContainer" containerID="49ae3dc706943e390d9805fe4150fc4e354d90c8c28f52d068f3e9c672c87b7d"
Sep 18 20:31:52.226431 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:31:52.226397    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ae3dc706943e390d9805fe4150fc4e354d90c8c28f52d068f3e9c672c87b7d\": container with ID starting with 49ae3dc706943e390d9805fe4150fc4e354d90c8c28f52d068f3e9c672c87b7d not found: ID does not exist" containerID="49ae3dc706943e390d9805fe4150fc4e354d90c8c28f52d068f3e9c672c87b7d"
Sep 18 20:31:52.226689 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.226431    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ae3dc706943e390d9805fe4150fc4e354d90c8c28f52d068f3e9c672c87b7d"} err="failed to get container status \"49ae3dc706943e390d9805fe4150fc4e354d90c8c28f52d068f3e9c672c87b7d\": rpc error: code = NotFound desc = could not find container \"49ae3dc706943e390d9805fe4150fc4e354d90c8c28f52d068f3e9c672c87b7d\": container with ID starting with 49ae3dc706943e390d9805fe4150fc4e354d90c8c28f52d068f3e9c672c87b7d not found: ID does not exist"
Sep 18 20:31:52.226689 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.226457    2578 scope.go:117] "RemoveContainer" containerID="2b20707e3ea13f570cc34fbfa5009739f6847e4a228099be388d3556d6596f4c"
Sep 18 20:31:52.234778 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.234752    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch" podUID="a12d929b-4cdd-4aeb-bc3e-b23dcce48b77" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.62:8443/health\": dial tcp 10.132.0.62:8443: connect: connection refused"
Sep 18 20:31:52.258230 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.258173    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62918655-ac3d-4e65-b7d1-84a1b797f5a9-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:31:52.258230 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.258194    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzznh\" (UniqueName: \"kubernetes.io/projected/440a5a06-5f73-40f5-b7d3-90b9f6ae925a-kube-api-access-vzznh\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:31:52.258230 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.258206    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/440a5a06-5f73-40f5-b7d3-90b9f6ae925a-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:31:52.258230 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.258215    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbhfn\" (UniqueName: \"kubernetes.io/projected/62918655-ac3d-4e65-b7d1-84a1b797f5a9-kube-api-access-xbhfn\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:31:52.298165 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.298139    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"]
Sep 18 20:31:52.300867 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.300832    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:52.306378 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.306357    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-40a1a5-7dd4f759c8-7c7tn"]
Sep 18 20:31:52.321395 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.321366    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62918655-ac3d-4e65-b7d1-84a1b797f5a9" path="/var/lib/kubelet/pods/62918655-ac3d-4e65-b7d1-84a1b797f5a9/volumes"
Sep 18 20:31:52.325491 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.325468    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"]
Sep 18 20:31:52.329720 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.329691    2578 scope.go:117] "RemoveContainer" containerID="2b20707e3ea13f570cc34fbfa5009739f6847e4a228099be388d3556d6596f4c"
Sep 18 20:31:52.330063 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:31:52.330040    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b20707e3ea13f570cc34fbfa5009739f6847e4a228099be388d3556d6596f4c\": container with ID starting with 2b20707e3ea13f570cc34fbfa5009739f6847e4a228099be388d3556d6596f4c not found: ID does not exist" containerID="2b20707e3ea13f570cc34fbfa5009739f6847e4a228099be388d3556d6596f4c"
Sep 18 20:31:52.330155 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.330070    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b20707e3ea13f570cc34fbfa5009739f6847e4a228099be388d3556d6596f4c"} err="failed to get container status \"2b20707e3ea13f570cc34fbfa5009739f6847e4a228099be388d3556d6596f4c\": rpc error: code = NotFound desc = could not find container \"2b20707e3ea13f570cc34fbfa5009739f6847e4a228099be388d3556d6596f4c\": container with ID starting with 2b20707e3ea13f570cc34fbfa5009739f6847e4a228099be388d3556d6596f4c not found: ID does not exist"
Sep 18 20:31:52.330463 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.330442    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-40a1a5-5675bc9fbf-7jjsc"]
Sep 18 20:31:52.449241 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.449209    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:31:52.561040 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.560968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-maas-api-tls\") pod \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") "
Sep 18 20:31:52.561040 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.561015    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-metrics-certs\") pod \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") "
Sep 18 20:31:52.561040 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.561038    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-openshift-service-ca\") pod \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") "
Sep 18 20:31:52.561265 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.561068    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnwtn\" (UniqueName: \"kubernetes.io/projected/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-kube-api-access-hnwtn\") pod \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\" (UID: \"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77\") "
Sep 18 20:31:52.561436 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.561406    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a12d929b-4cdd-4aeb-bc3e-b23dcce48b77" (UID: "a12d929b-4cdd-4aeb-bc3e-b23dcce48b77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:52.563173 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.563147    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a12d929b-4cdd-4aeb-bc3e-b23dcce48b77" (UID: "a12d929b-4cdd-4aeb-bc3e-b23dcce48b77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:31:52.563279 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.563188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a12d929b-4cdd-4aeb-bc3e-b23dcce48b77" (UID: "a12d929b-4cdd-4aeb-bc3e-b23dcce48b77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:31:52.563279 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.563204    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-kube-api-access-hnwtn" (OuterVolumeSpecName: "kube-api-access-hnwtn") pod "a12d929b-4cdd-4aeb-bc3e-b23dcce48b77" (UID: "a12d929b-4cdd-4aeb-bc3e-b23dcce48b77"). InnerVolumeSpecName "kube-api-access-hnwtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:52.629089 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.629052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:52.662287 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.662256    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:31:52.662287 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.662281    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:31:52.662463 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.662296    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:31:52.662463 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.662309    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnwtn\" (UniqueName: \"kubernetes.io/projected/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77-kube-api-access-hnwtn\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:31:52.778084 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:52.778049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:53.224122 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:53.224084    2578 generic.go:356] "Generic (PLEG): container finished" podID="a12d929b-4cdd-4aeb-bc3e-b23dcce48b77" containerID="2ed227f3a53c507daa0aec96011a5bdc82f95f97ab9e553f844ff062e7d573b2" exitCode=0
Sep 18 20:31:53.224308 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:53.224148    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"
Sep 18 20:31:53.224308 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:53.224173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch" event={"ID":"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77","Type":"ContainerDied","Data":"2ed227f3a53c507daa0aec96011a5bdc82f95f97ab9e553f844ff062e7d573b2"}
Sep 18 20:31:53.224308 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:53.224261    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch" event={"ID":"a12d929b-4cdd-4aeb-bc3e-b23dcce48b77","Type":"ContainerDied","Data":"54d48b637b79884b0c19318b4b4e41e93ae1d29275d2c8e1dcc1970bfba19561"}
Sep 18 20:31:53.224308 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:53.224289    2578 scope.go:117] "RemoveContainer" containerID="2ed227f3a53c507daa0aec96011a5bdc82f95f97ab9e553f844ff062e7d573b2"
Sep 18 20:31:53.233613 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:53.233596    2578 scope.go:117] "RemoveContainer" containerID="2ed227f3a53c507daa0aec96011a5bdc82f95f97ab9e553f844ff062e7d573b2"
Sep 18 20:31:53.233886 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:31:53.233867    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed227f3a53c507daa0aec96011a5bdc82f95f97ab9e553f844ff062e7d573b2\": container with ID starting with 2ed227f3a53c507daa0aec96011a5bdc82f95f97ab9e553f844ff062e7d573b2 not found: ID does not exist" containerID="2ed227f3a53c507daa0aec96011a5bdc82f95f97ab9e553f844ff062e7d573b2"
Sep 18 20:31:53.233952 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:53.233894    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed227f3a53c507daa0aec96011a5bdc82f95f97ab9e553f844ff062e7d573b2"} err="failed to get container status \"2ed227f3a53c507daa0aec96011a5bdc82f95f97ab9e553f844ff062e7d573b2\": rpc error: code = NotFound desc = could not find container \"2ed227f3a53c507daa0aec96011a5bdc82f95f97ab9e553f844ff062e7d573b2\": container with ID starting with 2ed227f3a53c507daa0aec96011a5bdc82f95f97ab9e553f844ff062e7d573b2 not found: ID does not exist"
Sep 18 20:31:53.245848 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:53.245826    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"]
Sep 18 20:31:53.250148 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:53.250126    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-40a1a5-7bf74d7594-8lrch"]
Sep 18 20:31:53.694151 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:53.694060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:54.317782 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:54.317749    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="440a5a06-5f73-40f5-b7d3-90b9f6ae925a" path="/var/lib/kubelet/pods/440a5a06-5f73-40f5-b7d3-90b9f6ae925a/volumes"
Sep 18 20:31:54.318191 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:54.318149    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a12d929b-4cdd-4aeb-bc3e-b23dcce48b77" path="/var/lib/kubelet/pods/a12d929b-4cdd-4aeb-bc3e-b23dcce48b77/volumes"
Sep 18 20:31:59.306473 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.306434    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:59.421911 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.421879    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:59.566610 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.566520    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:31:59.935352 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935271    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b"]
Sep 18 20:31:59.935586 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935574    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed0fd99-1aba-4808-adb4-f1f58c1d928c" containerName="maas-api"
Sep 18 20:31:59.935634 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935588    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed0fd99-1aba-4808-adb4-f1f58c1d928c" containerName="maas-api"
Sep 18 20:31:59.935634 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935599    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62918655-ac3d-4e65-b7d1-84a1b797f5a9" containerName="payload-pre-processing"
Sep 18 20:31:59.935634 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935605    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="62918655-ac3d-4e65-b7d1-84a1b797f5a9" containerName="payload-pre-processing"
Sep 18 20:31:59.935634 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935613    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eddcc836-2b2c-4ad1-a887-c7d060800e0a" containerName="payload-processing"
Sep 18 20:31:59.935634 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935618    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eddcc836-2b2c-4ad1-a887-c7d060800e0a" containerName="payload-processing"
Sep 18 20:31:59.935634 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935625    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440a5a06-5f73-40f5-b7d3-90b9f6ae925a" containerName="payload-processing"
Sep 18 20:31:59.935634 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935630    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="440a5a06-5f73-40f5-b7d3-90b9f6ae925a" containerName="payload-processing"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935639    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4576a16-b734-4e58-b0ee-db28bc243509" containerName="payload-pre-processing"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935644    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4576a16-b734-4e58-b0ee-db28bc243509" containerName="payload-pre-processing"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935653    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2efb612-e0c1-4358-a4be-02e86a64cff7" containerName="payload-processing"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935658    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2efb612-e0c1-4358-a4be-02e86a64cff7" containerName="payload-processing"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935665    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1f615d-a08a-4d0e-8901-e99b85015ffd" containerName="maas-api"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935670    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1f615d-a08a-4d0e-8901-e99b85015ffd" containerName="maas-api"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935676    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12d929b-4cdd-4aeb-bc3e-b23dcce48b77" containerName="maas-api"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935681    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12d929b-4cdd-4aeb-bc3e-b23dcce48b77" containerName="maas-api"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935690    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3411e0c-5d64-4293-9b44-0003b4d0d046" containerName="cleanup"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935695    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3411e0c-5d64-4293-9b44-0003b4d0d046" containerName="cleanup"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935743    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3411e0c-5d64-4293-9b44-0003b4d0d046" containerName="cleanup"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935750    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1f615d-a08a-4d0e-8901-e99b85015ffd" containerName="maas-api"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935757    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a12d929b-4cdd-4aeb-bc3e-b23dcce48b77" containerName="maas-api"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935763    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="62918655-ac3d-4e65-b7d1-84a1b797f5a9" containerName="payload-pre-processing"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935768    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4576a16-b734-4e58-b0ee-db28bc243509" containerName="payload-pre-processing"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935775    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ed0fd99-1aba-4808-adb4-f1f58c1d928c" containerName="maas-api"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935781    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2efb612-e0c1-4358-a4be-02e86a64cff7" containerName="payload-processing"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935786    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="440a5a06-5f73-40f5-b7d3-90b9f6ae925a" containerName="payload-processing"
Sep 18 20:31:59.935887 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.935793    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eddcc836-2b2c-4ad1-a887-c7d060800e0a" containerName="payload-processing"
Sep 18 20:31:59.939915 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.939893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b"
Sep 18 20:31:59.942871 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.942849    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bhssz\""
Sep 18 20:31:59.953730 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:31:59.953704    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b"]
Sep 18 20:32:00.024882 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:00.024843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b\" (UID: \"ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b"
Sep 18 20:32:00.125519 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:00.125422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b\" (UID: \"ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b"
Sep 18 20:32:00.126044 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:00.126023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b\" (UID: \"ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b"
Sep 18 20:32:00.226801 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:00.222217    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:00.249349 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:00.249311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b"
Sep 18 20:32:00.524562 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:00.524457    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b"]
Sep 18 20:32:00.527413 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:32:00.527380    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe4beb1_5c5e_4dc4_b30a_8687726eb1ad.slice/crio-64538ae90e27df2e3128a08b291e662cb8680e6fc2ae77542e8f04beba85a2b5 WatchSource:0}: Error finding container 64538ae90e27df2e3128a08b291e662cb8680e6fc2ae77542e8f04beba85a2b5: Status 404 returned error can't find the container with id 64538ae90e27df2e3128a08b291e662cb8680e6fc2ae77542e8f04beba85a2b5
Sep 18 20:32:01.265208 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:01.265176    2578 generic.go:356] "Generic (PLEG): container finished" podID="ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad" containerID="a0b43538d8a2ab00a2bcb9b746ba5c6f008567d434e8f4f1938011fabb244632" exitCode=0
Sep 18 20:32:01.265371 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:01.265241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b" event={"ID":"ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad","Type":"ContainerDied","Data":"a0b43538d8a2ab00a2bcb9b746ba5c6f008567d434e8f4f1938011fabb244632"}
Sep 18 20:32:01.265371 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:01.265279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b" event={"ID":"ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad","Type":"ContainerStarted","Data":"64538ae90e27df2e3128a08b291e662cb8680e6fc2ae77542e8f04beba85a2b5"}
Sep 18 20:32:01.673661 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:01.673628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:02.394472 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:02.394445    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b"
Sep 18 20:32:02.548414 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:02.548331    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad-maas-api-service-ca\") pod \"ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad\" (UID: \"ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad\") "
Sep 18 20:32:02.548674 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:02.548649    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad" (UID: "ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:02.590766 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:02.590732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:02.649108 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:02.649067    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad-maas-api-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:32:03.277298 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:03.277272    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b"
Sep 18 20:32:03.277701 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:03.277271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b" event={"ID":"ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad","Type":"ContainerDied","Data":"64538ae90e27df2e3128a08b291e662cb8680e6fc2ae77542e8f04beba85a2b5"}
Sep 18 20:32:03.277701 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:03.277389    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64538ae90e27df2e3128a08b291e662cb8680e6fc2ae77542e8f04beba85a2b5"
Sep 18 20:32:10.669841 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:10.669805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:10.707578 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:10.707543    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"]
Sep 18 20:32:10.707786 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:10.707757    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg" podUID="d1af5b7f-f6db-4ca8-9e41-0ed3820f750d" containerName="payload-processing" containerID="cri-o://22b0b01fb96e6f11468016d66bcedeeb2f027293a2ebc6087fe3a1c92781a5e0" gracePeriod=30
Sep 18 20:32:10.767676 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:10.767642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:10.872688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:10.872652    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:10.917085 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:10.917053    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b"]
Sep 18 20:32:10.920782 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:10.920719    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81f19d-vg69b"]
Sep 18 20:32:10.959490 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:10.959467    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"
Sep 18 20:32:10.967760 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:10.967733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:11.119201 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.119172    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8sfq\" (UniqueName: \"kubernetes.io/projected/d1af5b7f-f6db-4ca8-9e41-0ed3820f750d-kube-api-access-f8sfq\") pod \"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d\" (UID: \"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d\") "
Sep 18 20:32:11.119379 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.119266    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1af5b7f-f6db-4ca8-9e41-0ed3820f750d-plugins-config-volume\") pod \"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d\" (UID: \"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d\") "
Sep 18 20:32:11.119663 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.119641    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1af5b7f-f6db-4ca8-9e41-0ed3820f750d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1af5b7f-f6db-4ca8-9e41-0ed3820f750d" (UID: "d1af5b7f-f6db-4ca8-9e41-0ed3820f750d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:11.121353 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.121321    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1af5b7f-f6db-4ca8-9e41-0ed3820f750d-kube-api-access-f8sfq" (OuterVolumeSpecName: "kube-api-access-f8sfq") pod "d1af5b7f-f6db-4ca8-9e41-0ed3820f750d" (UID: "d1af5b7f-f6db-4ca8-9e41-0ed3820f750d"). InnerVolumeSpecName "kube-api-access-f8sfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:11.220065 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.220029    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1af5b7f-f6db-4ca8-9e41-0ed3820f750d-plugins-config-volume\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:32:11.220065 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.220063    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8sfq\" (UniqueName: \"kubernetes.io/projected/d1af5b7f-f6db-4ca8-9e41-0ed3820f750d-kube-api-access-f8sfq\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:32:11.296549 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.296511    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:11.314453 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.314424    2578 generic.go:356] "Generic (PLEG): container finished" podID="d1af5b7f-f6db-4ca8-9e41-0ed3820f750d" containerID="22b0b01fb96e6f11468016d66bcedeeb2f027293a2ebc6087fe3a1c92781a5e0" exitCode=0
Sep 18 20:32:11.314609 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.314461    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"
Sep 18 20:32:11.314609 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.314463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg" event={"ID":"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d","Type":"ContainerDied","Data":"22b0b01fb96e6f11468016d66bcedeeb2f027293a2ebc6087fe3a1c92781a5e0"}
Sep 18 20:32:11.314685 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.314605    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg" event={"ID":"d1af5b7f-f6db-4ca8-9e41-0ed3820f750d","Type":"ContainerDied","Data":"6a56c59c3331275f1d39c32c3377647d9e54e715e30285b09880463311316902"}
Sep 18 20:32:11.314685 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.314635    2578 scope.go:117] "RemoveContainer" containerID="22b0b01fb96e6f11468016d66bcedeeb2f027293a2ebc6087fe3a1c92781a5e0"
Sep 18 20:32:11.323180 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.323163    2578 scope.go:117] "RemoveContainer" containerID="22b0b01fb96e6f11468016d66bcedeeb2f027293a2ebc6087fe3a1c92781a5e0"
Sep 18 20:32:11.323448 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:32:11.323418    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b0b01fb96e6f11468016d66bcedeeb2f027293a2ebc6087fe3a1c92781a5e0\": container with ID starting with 22b0b01fb96e6f11468016d66bcedeeb2f027293a2ebc6087fe3a1c92781a5e0 not found: ID does not exist" containerID="22b0b01fb96e6f11468016d66bcedeeb2f027293a2ebc6087fe3a1c92781a5e0"
Sep 18 20:32:11.323526 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.323442    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b0b01fb96e6f11468016d66bcedeeb2f027293a2ebc6087fe3a1c92781a5e0"} err="failed to get container status \"22b0b01fb96e6f11468016d66bcedeeb2f027293a2ebc6087fe3a1c92781a5e0\": rpc error: code = NotFound desc = could not find container \"22b0b01fb96e6f11468016d66bcedeeb2f027293a2ebc6087fe3a1c92781a5e0\": container with ID starting with 22b0b01fb96e6f11468016d66bcedeeb2f027293a2ebc6087fe3a1c92781a5e0 not found: ID does not exist"
Sep 18 20:32:11.338561 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.338537    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"]
Sep 18 20:32:11.342717 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.342695    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-81f19d-669f594fb5-djwkg"]
Sep 18 20:32:11.393610 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:11.393582    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:12.323441 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:12.323407    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1af5b7f-f6db-4ca8-9e41-0ed3820f750d" path="/var/lib/kubelet/pods/d1af5b7f-f6db-4ca8-9e41-0ed3820f750d/volumes"
Sep 18 20:32:12.323854 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:12.323744    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad" path="/var/lib/kubelet/pods/ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad/volumes"
Sep 18 20:32:18.394307 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:18.394252    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:18.494431 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:18.494393    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:18.720767 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:18.720734    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:26.949730 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:26.949691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:32:27.031304 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:32:27.031272    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:45.627812 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:33:45.627778    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:33:47.174218 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:33:47.174180    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:35.768663 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:34:35.768624    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:35.849071 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:34:35.849035    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:57.883343 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:34:57.883307    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:34:57.960523 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:34:57.960478    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:08.872171 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:08.872131    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:08.963126 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:08.963089    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:40.642164 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:40.642130    2578 scope.go:117] "RemoveContainer" containerID="313359d340b8ce53ab95b642812d77974b5157b62b47fd4a417478ee68167ca6"
Sep 18 20:36:57.052070 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.051984    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:57.335142 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.335061    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"]
Sep 18 20:36:57.335375 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.335363    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad" containerName="revoke-keys"
Sep 18 20:36:57.335430 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.335376    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad" containerName="revoke-keys"
Sep 18 20:36:57.335430 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.335390    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1af5b7f-f6db-4ca8-9e41-0ed3820f750d" containerName="payload-processing"
Sep 18 20:36:57.335430 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.335396    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1af5b7f-f6db-4ca8-9e41-0ed3820f750d" containerName="payload-processing"
Sep 18 20:36:57.335545 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.335443    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1af5b7f-f6db-4ca8-9e41-0ed3820f750d" containerName="payload-processing"
Sep 18 20:36:57.335545 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.335452    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebe4beb1-5c5e-4dc4-b30a-8687726eb1ad" containerName="revoke-keys"
Sep 18 20:36:57.339616 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.339598    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.342336 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.342315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-37cb2f01-kserve-self-signed-certs\""
Sep 18 20:36:57.350688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.350663    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"]
Sep 18 20:36:57.362880 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.362846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:36:57.437900 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.437871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct7sz\" (UniqueName: \"kubernetes.io/projected/86093ebb-a6e2-406c-b9d8-8478f87f6203-kube-api-access-ct7sz\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.437900 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.437905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-model-cache\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.438156 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.437958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86093ebb-a6e2-406c-b9d8-8478f87f6203-tls-certs\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.438156 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.438047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-home\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.438156 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.438081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-tmp-dir\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.438156 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.438101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-dshm\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.538731 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.538695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-dshm\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.538915 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.538743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct7sz\" (UniqueName: \"kubernetes.io/projected/86093ebb-a6e2-406c-b9d8-8478f87f6203-kube-api-access-ct7sz\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.538915 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.538765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-model-cache\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.538915 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.538791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86093ebb-a6e2-406c-b9d8-8478f87f6203-tls-certs\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.538915 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.538838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-home\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.538915 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.538866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-tmp-dir\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.539282 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.539252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-model-cache\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.539406 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.539287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-tmp-dir\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.539406 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.539330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-home\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.541136 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.541108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-dshm\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.541309 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.541291    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86093ebb-a6e2-406c-b9d8-8478f87f6203-tls-certs\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.548380 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.548361    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct7sz\" (UniqueName: \"kubernetes.io/projected/86093ebb-a6e2-406c-b9d8-8478f87f6203-kube-api-access-ct7sz\") pod \"e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") " pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.648298 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.648222    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:36:57.934070 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:57.934047    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"]
Sep 18 20:36:57.935218 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:36:57.935191    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86093ebb_a6e2_406c_b9d8_8478f87f6203.slice/crio-99a3c130d5cccad3d94482dd9f23f1bbaa471639a2069fe7435753046180efba WatchSource:0}: Error finding container 99a3c130d5cccad3d94482dd9f23f1bbaa471639a2069fe7435753046180efba: Status 404 returned error can't find the container with id 99a3c130d5cccad3d94482dd9f23f1bbaa471639a2069fe7435753046180efba
Sep 18 20:36:58.649153 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:58.649117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx" event={"ID":"86093ebb-a6e2-406c-b9d8-8478f87f6203","Type":"ContainerStarted","Data":"e29501dd7cb5afd9cf837c6d86a2b0cd2047fb58414aace9b77c17b93c9f4964"}
Sep 18 20:36:58.649578 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:58.649168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx" event={"ID":"86093ebb-a6e2-406c-b9d8-8478f87f6203","Type":"ContainerStarted","Data":"99a3c130d5cccad3d94482dd9f23f1bbaa471639a2069fe7435753046180efba"}
Sep 18 20:36:58.673024 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:36:58.672974    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx" podStartSLOduration=1.6729600900000001 podStartE2EDuration="1.67296009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:36:58.671394973 +0000 UTC m=+2420.509858924" watchObservedRunningTime="2026-09-18 20:36:58.67296009 +0000 UTC m=+2420.511424109"
Sep 18 20:37:06.531468 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.531429    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"]
Sep 18 20:37:06.534721 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.534704    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.538229 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:37:06.538205    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" is forbidden: User \"system:node:ip-10-0-132-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-132-172.ec2.internal' and this object" podUID="fbe28f1a-b768-44eb-b172-ae9c945b587d" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.540691 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:37:06.540660    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-same-id-b-37cb2f01-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-132-172.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-132-172.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"e2e-same-id-b-37cb2f01-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 18 20:37:06.551966 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.551922    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"]
Sep 18 20:37:06.713404 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.713369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-home\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.713578 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.713424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-model-cache\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.713578 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.713472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe28f1a-b768-44eb-b172-ae9c945b587d-tls-certs\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.713578 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.713487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k75k\" (UniqueName: \"kubernetes.io/projected/fbe28f1a-b768-44eb-b172-ae9c945b587d-kube-api-access-7k75k\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.713578 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.713505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-tmp-dir\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.713578 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.713542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-dshm\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.814244 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.814157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-model-cache\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.814244 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.814204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe28f1a-b768-44eb-b172-ae9c945b587d-tls-certs\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.814244 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.814220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k75k\" (UniqueName: \"kubernetes.io/projected/fbe28f1a-b768-44eb-b172-ae9c945b587d-kube-api-access-7k75k\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.814244 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.814239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-tmp-dir\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.814595 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.814271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-dshm\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.814595 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.814296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-home\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.814712 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.814609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-home\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.814712 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.814613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-model-cache\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.814797 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.814709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-tmp-dir\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.816503 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.816485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-dshm\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:06.827232 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:06.827210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k75k\" (UniqueName: \"kubernetes.io/projected/fbe28f1a-b768-44eb-b172-ae9c945b587d-kube-api-access-7k75k\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:07.599504 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:07.599459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:37:07.648750 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:07.648715    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:37:07.648910 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:07.648781    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:37:07.661232 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:07.661209    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:37:07.701635 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:07.701613    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:37:07.814444 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:37:07.814410    2578 secret.go:189] Couldn't get secret llm/e2e-same-id-b-37cb2f01-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition
Sep 18 20:37:07.814605 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:37:07.814514    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe28f1a-b768-44eb-b172-ae9c945b587d-tls-certs podName:fbe28f1a-b768-44eb-b172-ae9c945b587d nodeName:}" failed. No retries permitted until 2026-09-18 20:37:08.314492497 +0000 UTC m=+2430.152956434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fbe28f1a-b768-44eb-b172-ae9c945b587d-tls-certs") pod "e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg" (UID: "fbe28f1a-b768-44eb-b172-ae9c945b587d") : failed to sync secret cache: timed out waiting for the condition
Sep 18 20:37:08.046508 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:08.046477    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-37cb2f01-kserve-self-signed-certs\""
Sep 18 20:37:08.328284 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:08.328193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe28f1a-b768-44eb-b172-ae9c945b587d-tls-certs\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:08.330734 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:08.330701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe28f1a-b768-44eb-b172-ae9c945b587d-tls-certs\") pod \"e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") " pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:08.343552 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:08.343530    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:08.623993 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:08.623907    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"]
Sep 18 20:37:08.627580 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:37:08.627552    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbe28f1a_b768_44eb_b172_ae9c945b587d.slice/crio-e2d3a2b1cb3396595aff0c0c47a11d2982027ce2af2dc777b2c260654461eddd WatchSource:0}: Error finding container e2d3a2b1cb3396595aff0c0c47a11d2982027ce2af2dc777b2c260654461eddd: Status 404 returned error can't find the container with id e2d3a2b1cb3396595aff0c0c47a11d2982027ce2af2dc777b2c260654461eddd
Sep 18 20:37:08.697090 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:08.697065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg" event={"ID":"fbe28f1a-b768-44eb-b172-ae9c945b587d","Type":"ContainerStarted","Data":"e2d3a2b1cb3396595aff0c0c47a11d2982027ce2af2dc777b2c260654461eddd"}
Sep 18 20:37:09.703163 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:09.703128    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg" event={"ID":"fbe28f1a-b768-44eb-b172-ae9c945b587d","Type":"ContainerStarted","Data":"816d99548562d5e6a333bd368f9297ed803fb46bfa20f2116d8ba78e29bf82de"}
Sep 18 20:37:09.746903 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:09.746848    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg" podStartSLOduration=3.746830964 podStartE2EDuration="3.746830964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:37:09.744571778 +0000 UTC m=+2431.583035729" watchObservedRunningTime="2026-09-18 20:37:09.746830964 +0000 UTC m=+2431.585294915"
Sep 18 20:37:12.705300 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:12.705255    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:37:18.344722 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:18.344690    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:18.347212 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:18.344735    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:18.356776 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:18.356754    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:18.756574 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:18.756541    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:26.612303 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:26.612264    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"]
Sep 18 20:37:26.640611 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:26.640578    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:37:27.841895 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:27.841862    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"]
Sep 18 20:37:27.867890 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:27.867860    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:37:41.630832 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:41.630788    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx" podUID="86093ebb-a6e2-406c-b9d8-8478f87f6203" containerName="main" containerID="cri-o://e29501dd7cb5afd9cf837c6d86a2b0cd2047fb58414aace9b77c17b93c9f4964" gracePeriod=45
Sep 18 20:37:41.852048 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:41.852018    2578 generic.go:356] "Generic (PLEG): container finished" podID="86093ebb-a6e2-406c-b9d8-8478f87f6203" containerID="e29501dd7cb5afd9cf837c6d86a2b0cd2047fb58414aace9b77c17b93c9f4964" exitCode=0
Sep 18 20:37:41.852204 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:41.852089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx" event={"ID":"86093ebb-a6e2-406c-b9d8-8478f87f6203","Type":"ContainerDied","Data":"e29501dd7cb5afd9cf837c6d86a2b0cd2047fb58414aace9b77c17b93c9f4964"}
Sep 18 20:37:41.874718 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:41.874696    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:37:42.017737 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.017691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-dshm\") pod \"86093ebb-a6e2-406c-b9d8-8478f87f6203\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") "
Sep 18 20:37:42.017906 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.017786    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct7sz\" (UniqueName: \"kubernetes.io/projected/86093ebb-a6e2-406c-b9d8-8478f87f6203-kube-api-access-ct7sz\") pod \"86093ebb-a6e2-406c-b9d8-8478f87f6203\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") "
Sep 18 20:37:42.017906 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.017812    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86093ebb-a6e2-406c-b9d8-8478f87f6203-tls-certs\") pod \"86093ebb-a6e2-406c-b9d8-8478f87f6203\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") "
Sep 18 20:37:42.017906 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.017861    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-model-cache\") pod \"86093ebb-a6e2-406c-b9d8-8478f87f6203\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") "
Sep 18 20:37:42.017906 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.017883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-home\") pod \"86093ebb-a6e2-406c-b9d8-8478f87f6203\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") "
Sep 18 20:37:42.018127 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.017915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-tmp-dir\") pod \"86093ebb-a6e2-406c-b9d8-8478f87f6203\" (UID: \"86093ebb-a6e2-406c-b9d8-8478f87f6203\") "
Sep 18 20:37:42.018217 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.018181    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-model-cache" (OuterVolumeSpecName: "model-cache") pod "86093ebb-a6e2-406c-b9d8-8478f87f6203" (UID: "86093ebb-a6e2-406c-b9d8-8478f87f6203"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:42.018280 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.018197    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-home" (OuterVolumeSpecName: "home") pod "86093ebb-a6e2-406c-b9d8-8478f87f6203" (UID: "86093ebb-a6e2-406c-b9d8-8478f87f6203"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:42.018330 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.018277    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86093ebb-a6e2-406c-b9d8-8478f87f6203" (UID: "86093ebb-a6e2-406c-b9d8-8478f87f6203"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:42.020158 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.020135    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86093ebb-a6e2-406c-b9d8-8478f87f6203-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86093ebb-a6e2-406c-b9d8-8478f87f6203" (UID: "86093ebb-a6e2-406c-b9d8-8478f87f6203"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:37:42.020245 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.020176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86093ebb-a6e2-406c-b9d8-8478f87f6203-kube-api-access-ct7sz" (OuterVolumeSpecName: "kube-api-access-ct7sz") pod "86093ebb-a6e2-406c-b9d8-8478f87f6203" (UID: "86093ebb-a6e2-406c-b9d8-8478f87f6203"). InnerVolumeSpecName "kube-api-access-ct7sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:37:42.020245 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.020193    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-dshm" (OuterVolumeSpecName: "dshm") pod "86093ebb-a6e2-406c-b9d8-8478f87f6203" (UID: "86093ebb-a6e2-406c-b9d8-8478f87f6203"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:42.118747 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.118714    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct7sz\" (UniqueName: \"kubernetes.io/projected/86093ebb-a6e2-406c-b9d8-8478f87f6203-kube-api-access-ct7sz\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:42.118747 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.118739    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86093ebb-a6e2-406c-b9d8-8478f87f6203-tls-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:42.118747 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.118749    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-model-cache\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:42.118747 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.118757    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-home\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:42.119038 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.118765    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-tmp-dir\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:42.119038 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.118772    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86093ebb-a6e2-406c-b9d8-8478f87f6203-dshm\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:42.859064 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.859035    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"
Sep 18 20:37:42.859544 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.859039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx" event={"ID":"86093ebb-a6e2-406c-b9d8-8478f87f6203","Type":"ContainerDied","Data":"99a3c130d5cccad3d94482dd9f23f1bbaa471639a2069fe7435753046180efba"}
Sep 18 20:37:42.859544 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.859169    2578 scope.go:117] "RemoveContainer" containerID="e29501dd7cb5afd9cf837c6d86a2b0cd2047fb58414aace9b77c17b93c9f4964"
Sep 18 20:37:42.863896 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.863864    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg" podUID="fbe28f1a-b768-44eb-b172-ae9c945b587d" containerName="main" containerID="cri-o://816d99548562d5e6a333bd368f9297ed803fb46bfa20f2116d8ba78e29bf82de" gracePeriod=45
Sep 18 20:37:42.885741 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.885718    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"]
Sep 18 20:37:42.893643 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:42.893617    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-37cb2f01-kserve-7d4dc6f74c-z2cwx"]
Sep 18 20:37:43.107228 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.107205    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:43.227660 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.227635    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k75k\" (UniqueName: \"kubernetes.io/projected/fbe28f1a-b768-44eb-b172-ae9c945b587d-kube-api-access-7k75k\") pod \"fbe28f1a-b768-44eb-b172-ae9c945b587d\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") "
Sep 18 20:37:43.227818 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.227685    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-tmp-dir\") pod \"fbe28f1a-b768-44eb-b172-ae9c945b587d\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") "
Sep 18 20:37:43.227818 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.227718    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe28f1a-b768-44eb-b172-ae9c945b587d-tls-certs\") pod \"fbe28f1a-b768-44eb-b172-ae9c945b587d\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") "
Sep 18 20:37:43.227818 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.227741    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-home\") pod \"fbe28f1a-b768-44eb-b172-ae9c945b587d\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") "
Sep 18 20:37:43.227818 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.227765    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-dshm\") pod \"fbe28f1a-b768-44eb-b172-ae9c945b587d\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") "
Sep 18 20:37:43.227818 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.227794    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-model-cache\") pod \"fbe28f1a-b768-44eb-b172-ae9c945b587d\" (UID: \"fbe28f1a-b768-44eb-b172-ae9c945b587d\") "
Sep 18 20:37:43.228085 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.228062    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-home" (OuterVolumeSpecName: "home") pod "fbe28f1a-b768-44eb-b172-ae9c945b587d" (UID: "fbe28f1a-b768-44eb-b172-ae9c945b587d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:43.228155 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.228096    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fbe28f1a-b768-44eb-b172-ae9c945b587d" (UID: "fbe28f1a-b768-44eb-b172-ae9c945b587d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:43.228256 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.228230    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-model-cache" (OuterVolumeSpecName: "model-cache") pod "fbe28f1a-b768-44eb-b172-ae9c945b587d" (UID: "fbe28f1a-b768-44eb-b172-ae9c945b587d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:43.229989 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.229967    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-dshm" (OuterVolumeSpecName: "dshm") pod "fbe28f1a-b768-44eb-b172-ae9c945b587d" (UID: "fbe28f1a-b768-44eb-b172-ae9c945b587d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:37:43.229989 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.229980    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe28f1a-b768-44eb-b172-ae9c945b587d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fbe28f1a-b768-44eb-b172-ae9c945b587d" (UID: "fbe28f1a-b768-44eb-b172-ae9c945b587d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:37:43.229989 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.229969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe28f1a-b768-44eb-b172-ae9c945b587d-kube-api-access-7k75k" (OuterVolumeSpecName: "kube-api-access-7k75k") pod "fbe28f1a-b768-44eb-b172-ae9c945b587d" (UID: "fbe28f1a-b768-44eb-b172-ae9c945b587d"). InnerVolumeSpecName "kube-api-access-7k75k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:37:43.328587 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.328556    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k75k\" (UniqueName: \"kubernetes.io/projected/fbe28f1a-b768-44eb-b172-ae9c945b587d-kube-api-access-7k75k\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:43.328587 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.328584    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-tmp-dir\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:43.328587 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.328596    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe28f1a-b768-44eb-b172-ae9c945b587d-tls-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:43.328804 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.328604    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-home\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:43.328804 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.328611    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-dshm\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:43.328804 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.328618    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe28f1a-b768-44eb-b172-ae9c945b587d-model-cache\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:43.864817 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.864780    2578 generic.go:356] "Generic (PLEG): container finished" podID="fbe28f1a-b768-44eb-b172-ae9c945b587d" containerID="816d99548562d5e6a333bd368f9297ed803fb46bfa20f2116d8ba78e29bf82de" exitCode=0
Sep 18 20:37:43.864817 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.864815    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"
Sep 18 20:37:43.865363 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.864861    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg" event={"ID":"fbe28f1a-b768-44eb-b172-ae9c945b587d","Type":"ContainerDied","Data":"816d99548562d5e6a333bd368f9297ed803fb46bfa20f2116d8ba78e29bf82de"}
Sep 18 20:37:43.865363 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.864902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg" event={"ID":"fbe28f1a-b768-44eb-b172-ae9c945b587d","Type":"ContainerDied","Data":"e2d3a2b1cb3396595aff0c0c47a11d2982027ce2af2dc777b2c260654461eddd"}
Sep 18 20:37:43.865363 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.864921    2578 scope.go:117] "RemoveContainer" containerID="816d99548562d5e6a333bd368f9297ed803fb46bfa20f2116d8ba78e29bf82de"
Sep 18 20:37:43.873959 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.873912    2578 scope.go:117] "RemoveContainer" containerID="816d99548562d5e6a333bd368f9297ed803fb46bfa20f2116d8ba78e29bf82de"
Sep 18 20:37:43.874699 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:37:43.874672    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"816d99548562d5e6a333bd368f9297ed803fb46bfa20f2116d8ba78e29bf82de\": container with ID starting with 816d99548562d5e6a333bd368f9297ed803fb46bfa20f2116d8ba78e29bf82de not found: ID does not exist" containerID="816d99548562d5e6a333bd368f9297ed803fb46bfa20f2116d8ba78e29bf82de"
Sep 18 20:37:43.874845 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.874821    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"816d99548562d5e6a333bd368f9297ed803fb46bfa20f2116d8ba78e29bf82de"} err="failed to get container status \"816d99548562d5e6a333bd368f9297ed803fb46bfa20f2116d8ba78e29bf82de\": rpc error: code = NotFound desc = could not find container \"816d99548562d5e6a333bd368f9297ed803fb46bfa20f2116d8ba78e29bf82de\": container with ID starting with 816d99548562d5e6a333bd368f9297ed803fb46bfa20f2116d8ba78e29bf82de not found: ID does not exist"
Sep 18 20:37:43.887061 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.887039    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"]
Sep 18 20:37:43.892058 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:43.892036    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-37cb2f01-kserve-554dd654f9-9rcfg"]
Sep 18 20:37:44.317471 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:44.317441    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86093ebb-a6e2-406c-b9d8-8478f87f6203" path="/var/lib/kubelet/pods/86093ebb-a6e2-406c-b9d8-8478f87f6203/volumes"
Sep 18 20:37:44.317831 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:44.317817    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbe28f1a-b768-44eb-b172-ae9c945b587d" path="/var/lib/kubelet/pods/fbe28f1a-b768-44eb-b172-ae9c945b587d/volumes"
Sep 18 20:37:46.854531 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.854495    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"]
Sep 18 20:37:46.854948 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.854821    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86093ebb-a6e2-406c-b9d8-8478f87f6203" containerName="main"
Sep 18 20:37:46.854948 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.854833    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="86093ebb-a6e2-406c-b9d8-8478f87f6203" containerName="main"
Sep 18 20:37:46.854948 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.854851    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe28f1a-b768-44eb-b172-ae9c945b587d" containerName="main"
Sep 18 20:37:46.854948 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.854856    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe28f1a-b768-44eb-b172-ae9c945b587d" containerName="main"
Sep 18 20:37:46.854948 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.854900    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="86093ebb-a6e2-406c-b9d8-8478f87f6203" containerName="main"
Sep 18 20:37:46.854948 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.854909    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe28f1a-b768-44eb-b172-ae9c945b587d" containerName="main"
Sep 18 20:37:46.858893 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.858875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:46.861217 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.861196    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8kcjh\""
Sep 18 20:37:46.862004 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.861988    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 20:37:46.862004 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.861994    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 20:37:46.866401 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.866374    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"]
Sep 18 20:37:46.869334 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.869319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:46.876791 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.876770    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"]
Sep 18 20:37:46.895269 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.895240    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"]
Sep 18 20:37:46.958079 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.958045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2-maas-api-tls\") pod \"maas-api-68c86dfd56-hx5zg\" (UID: \"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:46.958242 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.958095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7998b2a-e7d6-4951-a554-c79b6b0dad13-metrics-certs\") pod \"maas-api-5598d8c9fb-x5qk7\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:46.958242 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.958113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2-metrics-certs\") pod \"maas-api-68c86dfd56-hx5zg\" (UID: \"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:46.958242 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.958167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7998b2a-e7d6-4951-a554-c79b6b0dad13-maas-api-tls\") pod \"maas-api-5598d8c9fb-x5qk7\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:46.958242 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.958199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2-openshift-service-ca\") pod \"maas-api-68c86dfd56-hx5zg\" (UID: \"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:46.958242 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.958225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7998b2a-e7d6-4951-a554-c79b6b0dad13-openshift-service-ca\") pod \"maas-api-5598d8c9fb-x5qk7\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:46.958478 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.958253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbkb\" (UniqueName: \"kubernetes.io/projected/3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2-kube-api-access-qqbkb\") pod \"maas-api-68c86dfd56-hx5zg\" (UID: \"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:46.958478 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:46.958328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2wt4\" (UniqueName: \"kubernetes.io/projected/e7998b2a-e7d6-4951-a554-c79b6b0dad13-kube-api-access-x2wt4\") pod \"maas-api-5598d8c9fb-x5qk7\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:47.059459 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.059423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2-maas-api-tls\") pod \"maas-api-68c86dfd56-hx5zg\" (UID: \"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:47.059620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.059474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7998b2a-e7d6-4951-a554-c79b6b0dad13-metrics-certs\") pod \"maas-api-5598d8c9fb-x5qk7\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:47.059620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.059489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2-metrics-certs\") pod \"maas-api-68c86dfd56-hx5zg\" (UID: \"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:47.059620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.059506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7998b2a-e7d6-4951-a554-c79b6b0dad13-maas-api-tls\") pod \"maas-api-5598d8c9fb-x5qk7\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:47.059620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.059554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2-openshift-service-ca\") pod \"maas-api-68c86dfd56-hx5zg\" (UID: \"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:47.059620 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.059598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7998b2a-e7d6-4951-a554-c79b6b0dad13-openshift-service-ca\") pod \"maas-api-5598d8c9fb-x5qk7\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:47.059880 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.059626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbkb\" (UniqueName: \"kubernetes.io/projected/3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2-kube-api-access-qqbkb\") pod \"maas-api-68c86dfd56-hx5zg\" (UID: \"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:47.059880 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.059645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2wt4\" (UniqueName: \"kubernetes.io/projected/e7998b2a-e7d6-4951-a554-c79b6b0dad13-kube-api-access-x2wt4\") pod \"maas-api-5598d8c9fb-x5qk7\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:47.060419 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.060391    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7998b2a-e7d6-4951-a554-c79b6b0dad13-openshift-service-ca\") pod \"maas-api-5598d8c9fb-x5qk7\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:47.060518 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.060455    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2-openshift-service-ca\") pod \"maas-api-68c86dfd56-hx5zg\" (UID: \"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:47.062144 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.062124    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7998b2a-e7d6-4951-a554-c79b6b0dad13-maas-api-tls\") pod \"maas-api-5598d8c9fb-x5qk7\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:47.062229 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.062177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2-maas-api-tls\") pod \"maas-api-68c86dfd56-hx5zg\" (UID: \"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:47.062433 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.062409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7998b2a-e7d6-4951-a554-c79b6b0dad13-metrics-certs\") pod \"maas-api-5598d8c9fb-x5qk7\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:47.062532 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.062512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2-metrics-certs\") pod \"maas-api-68c86dfd56-hx5zg\" (UID: \"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:47.074396 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.074375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbkb\" (UniqueName: \"kubernetes.io/projected/3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2-kube-api-access-qqbkb\") pod \"maas-api-68c86dfd56-hx5zg\" (UID: \"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:47.077278 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.077261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2wt4\" (UniqueName: \"kubernetes.io/projected/e7998b2a-e7d6-4951-a554-c79b6b0dad13-kube-api-access-x2wt4\") pod \"maas-api-5598d8c9fb-x5qk7\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:47.169062 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.168972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:47.179179 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.179148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:47.456965 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.456919    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"]
Sep 18 20:37:47.459418 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:37:47.459392    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7998b2a_e7d6_4951_a554_c79b6b0dad13.slice/crio-0cc70e46da631460370e3c0bff04b8936c6cbed30451c8de8ad2379156669a52 WatchSource:0}: Error finding container 0cc70e46da631460370e3c0bff04b8936c6cbed30451c8de8ad2379156669a52: Status 404 returned error can't find the container with id 0cc70e46da631460370e3c0bff04b8936c6cbed30451c8de8ad2379156669a52
Sep 18 20:37:47.460758 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.460739    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:37:47.466440 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.466418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"]
Sep 18 20:37:47.466773 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:37:47.466754    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db8953e_bbd6_40f8_8c27_ab8fdc3fcdf2.slice/crio-79340dbffae1c41528e74d80785d87889d997f03aa4e9ac876269aac15fae2f0 WatchSource:0}: Error finding container 79340dbffae1c41528e74d80785d87889d997f03aa4e9ac876269aac15fae2f0: Status 404 returned error can't find the container with id 79340dbffae1c41528e74d80785d87889d997f03aa4e9ac876269aac15fae2f0
Sep 18 20:37:47.888511 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.888482    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7" event={"ID":"e7998b2a-e7d6-4951-a554-c79b6b0dad13","Type":"ContainerStarted","Data":"0cc70e46da631460370e3c0bff04b8936c6cbed30451c8de8ad2379156669a52"}
Sep 18 20:37:47.889812 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:47.889788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg" event={"ID":"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2","Type":"ContainerStarted","Data":"79340dbffae1c41528e74d80785d87889d997f03aa4e9ac876269aac15fae2f0"}
Sep 18 20:37:48.896355 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:48.896317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7" event={"ID":"e7998b2a-e7d6-4951-a554-c79b6b0dad13","Type":"ContainerStarted","Data":"f68e5d4b0edb67888c09ddd788f3c67d02b078ce404939c0a41c009e23a504be"}
Sep 18 20:37:48.896819 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:48.896409    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:48.898006 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:48.897983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg" event={"ID":"3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2","Type":"ContainerStarted","Data":"3ff3da986faf1f09ee55082718d01c051e63dd63bef179e0af9e76c069af7c19"}
Sep 18 20:37:48.898115 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:48.898027    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:48.921087 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:48.921046    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7" podStartSLOduration=2.539902348 podStartE2EDuration="2.921031978s" totalImagesPullingTime="381.12963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:37:48.918172616 +0000 UTC m=+2470.756636567" watchObservedRunningTime="2026-09-18 20:37:48.921031978 +0000 UTC m=+2470.759495929"
Sep 18 20:37:48.939358 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:48.939303    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg" podStartSLOduration=2.545339312 podStartE2EDuration="2.939291382s" totalImagesPullingTime="393.95207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:37:48.937561909 +0000 UTC m=+2470.776025875" watchObservedRunningTime="2026-09-18 20:37:48.939291382 +0000 UTC m=+2470.777755335"
Sep 18 20:37:54.908349 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:54.908320    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:54.908799 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:54.908772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-hx5zg"
Sep 18 20:37:54.990918 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:54.990886    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"]
Sep 18 20:37:54.991134 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:54.991074    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7" podUID="e7998b2a-e7d6-4951-a554-c79b6b0dad13" containerName="maas-api" containerID="cri-o://f68e5d4b0edb67888c09ddd788f3c67d02b078ce404939c0a41c009e23a504be" gracePeriod=30
Sep 18 20:37:55.231100 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.231076    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:55.330550 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.330514    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7998b2a-e7d6-4951-a554-c79b6b0dad13-maas-api-tls\") pod \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") "
Sep 18 20:37:55.330743 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.330559    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7998b2a-e7d6-4951-a554-c79b6b0dad13-openshift-service-ca\") pod \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") "
Sep 18 20:37:55.330743 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.330590    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7998b2a-e7d6-4951-a554-c79b6b0dad13-metrics-certs\") pod \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") "
Sep 18 20:37:55.330743 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.330719    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2wt4\" (UniqueName: \"kubernetes.io/projected/e7998b2a-e7d6-4951-a554-c79b6b0dad13-kube-api-access-x2wt4\") pod \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\" (UID: \"e7998b2a-e7d6-4951-a554-c79b6b0dad13\") "
Sep 18 20:37:55.330977 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.330924    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7998b2a-e7d6-4951-a554-c79b6b0dad13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e7998b2a-e7d6-4951-a554-c79b6b0dad13" (UID: "e7998b2a-e7d6-4951-a554-c79b6b0dad13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:37:55.331055 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.331024    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7998b2a-e7d6-4951-a554-c79b6b0dad13-openshift-service-ca\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:55.332863 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.332833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7998b2a-e7d6-4951-a554-c79b6b0dad13-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e7998b2a-e7d6-4951-a554-c79b6b0dad13" (UID: "e7998b2a-e7d6-4951-a554-c79b6b0dad13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:37:55.332863 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.332856    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7998b2a-e7d6-4951-a554-c79b6b0dad13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e7998b2a-e7d6-4951-a554-c79b6b0dad13" (UID: "e7998b2a-e7d6-4951-a554-c79b6b0dad13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:37:55.333059 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.332892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7998b2a-e7d6-4951-a554-c79b6b0dad13-kube-api-access-x2wt4" (OuterVolumeSpecName: "kube-api-access-x2wt4") pod "e7998b2a-e7d6-4951-a554-c79b6b0dad13" (UID: "e7998b2a-e7d6-4951-a554-c79b6b0dad13"). InnerVolumeSpecName "kube-api-access-x2wt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:37:55.431544 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.431509    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7998b2a-e7d6-4951-a554-c79b6b0dad13-metrics-certs\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:55.431544 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.431539    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2wt4\" (UniqueName: \"kubernetes.io/projected/e7998b2a-e7d6-4951-a554-c79b6b0dad13-kube-api-access-x2wt4\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:55.431544 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.431550    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7998b2a-e7d6-4951-a554-c79b6b0dad13-maas-api-tls\") on node \"ip-10-0-132-172.ec2.internal\" DevicePath \"\""
Sep 18 20:37:55.934462 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.934425    2578 generic.go:356] "Generic (PLEG): container finished" podID="e7998b2a-e7d6-4951-a554-c79b6b0dad13" containerID="f68e5d4b0edb67888c09ddd788f3c67d02b078ce404939c0a41c009e23a504be" exitCode=0
Sep 18 20:37:55.934892 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.934467    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"
Sep 18 20:37:55.934892 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.934506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7" event={"ID":"e7998b2a-e7d6-4951-a554-c79b6b0dad13","Type":"ContainerDied","Data":"f68e5d4b0edb67888c09ddd788f3c67d02b078ce404939c0a41c009e23a504be"}
Sep 18 20:37:55.934892 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.934555    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7" event={"ID":"e7998b2a-e7d6-4951-a554-c79b6b0dad13","Type":"ContainerDied","Data":"0cc70e46da631460370e3c0bff04b8936c6cbed30451c8de8ad2379156669a52"}
Sep 18 20:37:55.934892 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.934577    2578 scope.go:117] "RemoveContainer" containerID="f68e5d4b0edb67888c09ddd788f3c67d02b078ce404939c0a41c009e23a504be"
Sep 18 20:37:55.943497 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.943482    2578 scope.go:117] "RemoveContainer" containerID="f68e5d4b0edb67888c09ddd788f3c67d02b078ce404939c0a41c009e23a504be"
Sep 18 20:37:55.943762 ip-10-0-132-172 kubenswrapper[2578]: E0918 20:37:55.943744    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68e5d4b0edb67888c09ddd788f3c67d02b078ce404939c0a41c009e23a504be\": container with ID starting with f68e5d4b0edb67888c09ddd788f3c67d02b078ce404939c0a41c009e23a504be not found: ID does not exist" containerID="f68e5d4b0edb67888c09ddd788f3c67d02b078ce404939c0a41c009e23a504be"
Sep 18 20:37:55.943800 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.943773    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68e5d4b0edb67888c09ddd788f3c67d02b078ce404939c0a41c009e23a504be"} err="failed to get container status \"f68e5d4b0edb67888c09ddd788f3c67d02b078ce404939c0a41c009e23a504be\": rpc error: code = NotFound desc = could not find container \"f68e5d4b0edb67888c09ddd788f3c67d02b078ce404939c0a41c009e23a504be\": container with ID starting with f68e5d4b0edb67888c09ddd788f3c67d02b078ce404939c0a41c009e23a504be not found: ID does not exist"
Sep 18 20:37:55.958695 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.958669    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"]
Sep 18 20:37:55.962096 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:55.962074    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-x5qk7"]
Sep 18 20:37:56.323628 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:37:56.323585    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7998b2a-e7d6-4951-a554-c79b6b0dad13" path="/var/lib/kubelet/pods/e7998b2a-e7d6-4951-a554-c79b6b0dad13/volumes"
Sep 18 20:38:25.064484 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:25.064447    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:25.213772 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:25.213735    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:26.159644 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:26.159612    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wr6h9_dcf7a688-09a8-4a24-8931-6692efa454b6/manager/0.log"
Sep 18 20:38:26.927329 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:26.927300    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f76f79b89-cv5qp_32db5bf8-5c32-4712-abef-7f1cf31b5561/copy-manifests/0.log"
Sep 18 20:38:26.934460 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:26.934434    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f76f79b89-cv5qp_32db5bf8-5c32-4712-abef-7f1cf31b5561/manager/0.log"
Sep 18 20:38:27.398948 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:27.398913    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68c86dfd56-hx5zg_3db8953e-bbd6-40f8-8c27-ab8fdc3fcdf2/maas-api/0.log"
Sep 18 20:38:27.861058 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:27.861023    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-gf5ph_f376959c-a49d-405b-afcb-c8deec84a921/postgres/0.log"
Sep 18 20:38:28.740182 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:28.740143    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-65cnn_328ce793-8384-410d-9569-7adb784fbf13/manager/0.log"
Sep 18 20:38:29.418040 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:29.418007    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-9lcvw_076f3e26-3618-4db3-84ff-6c07b5d336b1/manager/0.log"
Sep 18 20:38:29.900836 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:29.900788    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j8kd5_807e886a-5281-4441-9363-a16a16855842/discovery/0.log"
Sep 18 20:38:30.018375 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:30.018337    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dc98cf6f8-p6m5k_66fe211b-0f00-4ee4-b678-9224e74a144c/kube-auth-proxy/0.log"
Sep 18 20:38:30.957106 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:30.957025    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jpkc4_3aeeb490-c889-4d39-8adf-85c2bae6c12b/main/0.log"
Sep 18 20:38:31.299235 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:31.299212    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-gdbqn_73b68fca-6f68-4e8d-8e44-72f731e40741/main/0.log"
Sep 18 20:38:31.420087 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:31.420052    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc9q48_02e801ca-602a-4eff-8119-00f494312bcc/main/0.log"
Sep 18 20:38:37.605596 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:37.605553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:37.706593 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:37.706547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log" limit=1024
Sep 18 20:38:40.669740 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:40.669709    2578 scope.go:117] "RemoveContainer" containerID="a0b43538d8a2ab00a2bcb9b746ba5c6f008567d434e8f4f1938011fabb244632"
Sep 18 20:38:40.880542 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:40.880512    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mshsc_ea005810-7434-414a-8805-b1a2032e8108/global-pull-secret-syncer/0.log"
Sep 18 20:38:41.058706 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:41.058676    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8bps7_9cd771fc-4b1c-49ca-a5b7-b31be69f17fd/konnectivity-agent/0.log"
Sep 18 20:38:41.209668 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:41.209638    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-172.ec2.internal_afec21724a7e77cd26c664aec35ac17d/haproxy/0.log"
Sep 18 20:38:45.739616 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:45.739585    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-65cnn_328ce793-8384-410d-9569-7adb784fbf13/manager/0.log"
Sep 18 20:38:45.973318 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:45.973286    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-9lcvw_076f3e26-3618-4db3-84ff-6c07b5d336b1/manager/0.log"
Sep 18 20:38:48.044181 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:48.044124    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7pfwf_b0a00281-39c2-4082-a8aa-8f5f4703d1ab/node-exporter/0.log"
Sep 18 20:38:48.068295 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:48.068268    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7pfwf_b0a00281-39c2-4082-a8aa-8f5f4703d1ab/kube-rbac-proxy/0.log"
Sep 18 20:38:48.093136 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:48.093114    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7pfwf_b0a00281-39c2-4082-a8aa-8f5f4703d1ab/init-textfile/0.log"
Sep 18 20:38:50.555451 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.555421    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"]
Sep 18 20:38:50.555831 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.555725    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7998b2a-e7d6-4951-a554-c79b6b0dad13" containerName="maas-api"
Sep 18 20:38:50.555831 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.555741    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7998b2a-e7d6-4951-a554-c79b6b0dad13" containerName="maas-api"
Sep 18 20:38:50.555831 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.555805    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7998b2a-e7d6-4951-a554-c79b6b0dad13" containerName="maas-api"
Sep 18 20:38:50.560007 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.559990    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.562754 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.562732    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w4xpn\"/\"openshift-service-ca.crt\""
Sep 18 20:38:50.562877 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.562819    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w4xpn\"/\"default-dockercfg-68gcr\""
Sep 18 20:38:50.563250 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.563235    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w4xpn\"/\"kube-root-ca.crt\""
Sep 18 20:38:50.572676 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.572650    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"]
Sep 18 20:38:50.675227 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.675191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/503abb1e-5dff-4807-b62e-72e433b0cbc9-podres\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.675391 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.675312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/503abb1e-5dff-4807-b62e-72e433b0cbc9-proc\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.675391 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.675373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/503abb1e-5dff-4807-b62e-72e433b0cbc9-lib-modules\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.675470 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.675399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgh7\" (UniqueName: \"kubernetes.io/projected/503abb1e-5dff-4807-b62e-72e433b0cbc9-kube-api-access-khgh7\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.675470 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.675433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/503abb1e-5dff-4807-b62e-72e433b0cbc9-sys\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.776437 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.776400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/503abb1e-5dff-4807-b62e-72e433b0cbc9-lib-modules\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.776437 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.776443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgh7\" (UniqueName: \"kubernetes.io/projected/503abb1e-5dff-4807-b62e-72e433b0cbc9-kube-api-access-khgh7\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.776688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.776460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/503abb1e-5dff-4807-b62e-72e433b0cbc9-sys\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.776688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.776480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/503abb1e-5dff-4807-b62e-72e433b0cbc9-podres\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.776688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.776563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/503abb1e-5dff-4807-b62e-72e433b0cbc9-sys\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.776688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.776570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/503abb1e-5dff-4807-b62e-72e433b0cbc9-proc\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.776688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.776579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/503abb1e-5dff-4807-b62e-72e433b0cbc9-lib-modules\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.776688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.776618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/503abb1e-5dff-4807-b62e-72e433b0cbc9-proc\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.776688 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.776678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/503abb1e-5dff-4807-b62e-72e433b0cbc9-podres\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.789046 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.789016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgh7\" (UniqueName: \"kubernetes.io/projected/503abb1e-5dff-4807-b62e-72e433b0cbc9-kube-api-access-khgh7\") pod \"perf-node-gather-daemonset-mz7dq\" (UID: \"503abb1e-5dff-4807-b62e-72e433b0cbc9\") " pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:50.876671 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:50.876596    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:51.159060 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:51.159032    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"]
Sep 18 20:38:51.161206 ip-10-0-132-172 kubenswrapper[2578]: W0918 20:38:51.161179    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod503abb1e_5dff_4807_b62e_72e433b0cbc9.slice/crio-011bb1f0d29e6e3237443e8c0dd30cff9612e5b09c4225cdec594cee38b19ed3 WatchSource:0}: Error finding container 011bb1f0d29e6e3237443e8c0dd30cff9612e5b09c4225cdec594cee38b19ed3: Status 404 returned error can't find the container with id 011bb1f0d29e6e3237443e8c0dd30cff9612e5b09c4225cdec594cee38b19ed3
Sep 18 20:38:51.190835 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:51.190804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq" event={"ID":"503abb1e-5dff-4807-b62e-72e433b0cbc9","Type":"ContainerStarted","Data":"011bb1f0d29e6e3237443e8c0dd30cff9612e5b09c4225cdec594cee38b19ed3"}
Sep 18 20:38:52.197656 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:52.197619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq" event={"ID":"503abb1e-5dff-4807-b62e-72e433b0cbc9","Type":"ContainerStarted","Data":"8b085ede07f082e5c095dac0368ea28377a34cbd4b61d24d1a71886b85e72e81"}
Sep 18 20:38:52.198207 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:52.197684    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:52.217357 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:52.217313    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq" podStartSLOduration=2.217297776 podStartE2EDuration="2.217297776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:38:52.217148922 +0000 UTC m=+2534.055612871" watchObservedRunningTime="2026-09-18 20:38:52.217297776 +0000 UTC m=+2534.055761774"
Sep 18 20:38:53.687567 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:53.687535    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9h4s5_6a9cb99c-d1fd-46f8-ae30-981161eaf053/dns/0.log"
Sep 18 20:38:53.716914 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:53.716884    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9h4s5_6a9cb99c-d1fd-46f8-ae30-981161eaf053/kube-rbac-proxy/0.log"
Sep 18 20:38:53.907739 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:53.907711    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fhfr4_a5470bc7-569a-490e-8ac3-e63b5738b859/dns-node-resolver/0.log"
Sep 18 20:38:54.536277 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:54.536251    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7shwk_1685cb44-5bb4-493c-b414-cb4e2ec0ab2f/node-ca/0.log"
Sep 18 20:38:56.281787 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:56.281755    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j8kd5_807e886a-5281-4441-9363-a16a16855842/discovery/0.log"
Sep 18 20:38:56.314081 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:56.314055    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dc98cf6f8-p6m5k_66fe211b-0f00-4ee4-b678-9224e74a144c/kube-auth-proxy/0.log"
Sep 18 20:38:57.800842 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:57.800802    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9pkl9_6aff7ff3-e6a6-4db7-8fc9-e21503f5ad23/serve-healthcheck-canary/0.log"
Sep 18 20:38:58.211074 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:58.211003    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w4xpn/perf-node-gather-daemonset-mz7dq"
Sep 18 20:38:58.537521 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:58.537484    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wxpxl_620c86f5-7746-491c-bc47-128d734e350f/kube-rbac-proxy/0.log"
Sep 18 20:38:58.562285 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:58.562253    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wxpxl_620c86f5-7746-491c-bc47-128d734e350f/exporter/0.log"
Sep 18 20:38:58.586253 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:38:58.586228    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wxpxl_620c86f5-7746-491c-bc47-128d734e350f/extractor/0.log"
Sep 18 20:39:01.237341 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:39:01.237317    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wr6h9_dcf7a688-09a8-4a24-8931-6692efa454b6/manager/0.log"
Sep 18 20:39:01.563799 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:39:01.563710    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j8kd5_807e886a-5281-4441-9363-a16a16855842/discovery/0.log"
Sep 18 20:39:01.592030 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:39:01.592006    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dc98cf6f8-p6m5k_66fe211b-0f00-4ee4-b678-9224e74a144c/kube-auth-proxy/0.log"
Sep 18 20:39:01.945922 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:39:01.945827    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f76f79b89-cv5qp_32db5bf8-5c32-4712-abef-7f1cf31b5561/manager/0.log"
Sep 18 20:39:01.987838 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:39:01.987813    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f76f79b89-cv5qp_32db5bf8-5c32-4712-abef-7f1cf31b5561/copy-manifests/0.log"
Sep 18 20:39:04.154768 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:39:04.154735    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j8kd5_807e886a-5281-4441-9363-a16a16855842/discovery/0.log"
Sep 18 20:39:04.203172 ip-10-0-132-172 kubenswrapper[2578]: I0918 20:39:04.203145    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dc98cf6f8-p6m5k_66fe211b-0f00-4ee4-b678-9224e74a144c/kube-auth-proxy/0.log"