Sep 18 21:06:02.279911 ip-10-0-130-92 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 21:06:02.688097 ip-10-0-130-92 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 21:06:02.688097 ip-10-0-130-92 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 21:06:02.688097 ip-10-0-130-92 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 21:06:02.688097 ip-10-0-130-92 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 21:06:02.688097 ip-10-0-130-92 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 21:06:02.689804 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.689740    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 21:06:02.694157 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694143    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:06:02.694157 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694157    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694161    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694165    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694168    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694171    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694173    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694176    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694179    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694182    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694184    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694187    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694190    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694196    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694199    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694201    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694204    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694206    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694209    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694211    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:06:02.694219 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694214    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694218    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694221    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694225    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694228    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694230    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694234    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694236    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694239    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694241    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694244    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694247    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694249    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694252    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694254    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694256    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694259    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694261    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694264    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694266    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:06:02.694682 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694269    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694271    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694273    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694276    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694278    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694281    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694284    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694286    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694288    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694293    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694297    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694300    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694302    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694305    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694308    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694311    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694314    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694316    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694319    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:06:02.695192 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694321    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694324    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694326    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694329    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694331    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694334    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694336    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694339    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694341    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694344    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694346    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694350    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694354    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694357    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694359    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694362    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694365    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694367    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694370    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:06:02.695658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694373    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694375    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694377    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694380    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694382    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694385    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694388    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694391    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694394    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694397    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694399    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694402    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694405    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694407    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694410    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694412    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694414    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694417    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694420    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694423    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:06:02.696123 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.694425    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695140    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695146    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695150    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695153    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695155    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695158    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695160    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695164    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695166    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695169    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695171    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695174    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695177    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695179    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695182    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695184    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695187    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695189    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695193    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:06:02.696635 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695196    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695199    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695201    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695204    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695207    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695209    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695212    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695215    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695217    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695220    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695222    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695225    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695228    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695230    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695233    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695235    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695238    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695240    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695242    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695245    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:06:02.697180 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695248    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695250    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695253    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695256    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695258    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695261    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695264    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695268    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695271    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695274    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695277    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695280    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695283    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695286    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695288    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695291    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695294    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695298    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695300    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:06:02.697697 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695303    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695306    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695309    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695311    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695313    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695316    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695318    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695321    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695323    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695326    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695329    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695331    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695334    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695336    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695340    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695342    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695345    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695348    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695350    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695352    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:06:02.698169 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695356    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695359    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695362    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695364    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695367    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695370    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695372    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695376    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695378    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695381    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695385    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695388    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695391    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695393    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695396    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695399    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695401    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695404    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695406    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:06:02.698673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695409    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695412    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695766    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695771    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695774    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695776    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695780    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695782    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695785    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695788    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695791    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695794    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695796    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695800    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695803    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695806    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695808    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695811    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695813    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:06:02.699133 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695816    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695819    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695821    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695824    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695827    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695829    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695832    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695836    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695839    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695841    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695844    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695847    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695850    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695853    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695855    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695857    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695860    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695863    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695865    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695868    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:06:02.699614 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695870    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695873    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695875    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695878    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695880    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695883    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695886    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695889    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695892    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695895    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695897    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695899    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695902    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695905    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695907    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695910    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695913    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695917    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695920    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695923    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:06:02.700110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695926    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695929    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695931    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695934    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695936    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695939    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695941    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695944    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695947    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695949    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695952    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695954    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695957    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695959    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695962    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695964    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695967    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695969    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695972    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695974    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:06:02.700599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695978    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695980    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695983    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695985    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695991    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695994    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.695997    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696000    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696003    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696006    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696008    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696012    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696015    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696017    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696020    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696022    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696025    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696028    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696031    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:06:02.701099 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696035    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696038    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696041    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696108    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696143    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696149    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696153    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696157    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696160    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696164    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696168    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696171    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696174    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696178    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696182    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696186    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696189    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696192    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696195    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696198    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696200    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696205    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696208    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696211    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 21:06:02.701567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696214    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696218    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696222    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696225    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696232    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696235    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696238    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696241    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696244    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696247    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696250    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696254    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696257    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696260    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696263    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696266    2579 flags.go:64] FLAG: --enable-server="true"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696269    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696273    2579 flags.go:64] FLAG: --event-burst="100"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696276    2579 flags.go:64] FLAG: --event-qps="50"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696279    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696282    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696285    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696288    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696292    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696295    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 21:06:02.702182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696298    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696300    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696303    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696306    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696309    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696311    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696314    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696317    2579 flags.go:64] FLAG: --feature-gates=""
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696321    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696324    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696327    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696330    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696333    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696335    2579 flags.go:64] FLAG: --help="false"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696338    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-130-92.ec2.internal"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696341    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696344    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696347    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696350    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696353    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696356    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696359    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696361    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 21:06:02.702870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696364    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696367    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696370    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696373    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696376    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696378    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696381    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696384    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696387    2579 flags.go:64] FLAG: --lock-file=""
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696390    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696393    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696396    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696403    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696406    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696409    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696411    2579 flags.go:64] FLAG: --logging-format="text"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696414    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696418    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696420    2579 flags.go:64] FLAG: --manifest-url=""
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696423    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696427    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696430    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696433    2579 flags.go:64] FLAG: --max-pods="110"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696436    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696439    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 21:06:02.703463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696442    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696445    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696448    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696451    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696454    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696461    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696464    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696467    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696470    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696473    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696478    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696481    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696484    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696487    2579 flags.go:64] FLAG: --port="10250"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696490    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696493    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eb8a43b5cde5a0ea"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696497    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696500    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696503    2579 flags.go:64] FLAG: --register-node="true"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696506    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696509    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696512    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696515    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696518    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696521    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 21:06:02.704379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696524    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696527    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696530    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696533    2579 flags.go:64] FLAG: --runonce="false"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696535    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696538    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696541    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696544    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696547    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696549    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696552    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696555    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696558    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696561    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696564    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696567    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696570    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696572    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696575    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696580    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696583    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696585    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696590    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696592    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696596    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 21:06:02.705032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696598    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696601    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696604    2579 flags.go:64] FLAG: --v="2"
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696608    2579 flags.go:64] FLAG: --version="false"
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696613    2579 flags.go:64] FLAG: --vmodule=""
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696617    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.696620    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696734    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696738    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696741    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696744    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696747    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696749    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696752    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696755    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696758    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696760    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696763    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696766    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696768    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696771    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:06:02.705663 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696775    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696778    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696780    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696783    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696785    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696788    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696790    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696793    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696795    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696798    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696800    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696804    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696807    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696809    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696812    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696815    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696818    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696821    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696823    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:06:02.706275 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696826    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696830    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696833    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696836    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696839    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696850    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696853    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696856    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696859    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696862    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696865    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696868    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696870    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696876    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696879    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696881    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696884    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696887    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696889    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696892    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:06:02.706815 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696895    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696897    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696900    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696902    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696905    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696908    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696910    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696913    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696919    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696922    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696925    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696928    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696931    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696934    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696937    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696940    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696943    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696945    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696949    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696951    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:06:02.707309 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696954    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696957    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696960    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696962    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696965    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696969    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696971    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696974    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696977    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696979    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696982    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696985    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696987    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696990    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696992    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696995    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.696997    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.697001    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.697008    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:06:02.707845 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.697011    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.697015    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.697017    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.697020    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.697023    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.697026    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.697029    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.697671    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.705541    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.705556    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.706532    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.706543    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706599    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706604    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:06:02.708302 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706608    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706611    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706614    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706616    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706619    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706622    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706626    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706630    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706633    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706637    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706653    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706657    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706660    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706663    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706668    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706672    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706675    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706678    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:06:02.708679 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706680    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706697    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706701    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706705    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706708    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706711    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706714    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706717    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706719    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706723    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706726    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706729    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706732    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706735    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706737    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706740    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706743    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706745    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706748    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706750    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:06:02.709117 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706753    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706756    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706758    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706761    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706763    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706766    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706768    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706771    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706774    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706776    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706779    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706781    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706784    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706787    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706789    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706792    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706794    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706797    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706799    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706802    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:06:02.709616 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706804    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706807    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706810    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706813    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706815    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706818    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706821    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706824    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706827    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706829    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706832    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706834    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706837    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706839    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706842    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706844    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706847    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706849    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706852    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706855    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:06:02.710111 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706858    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706861    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706870    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706874    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706877    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706880    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706883    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706885    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706888    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706891    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706893    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706896    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706898    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706901    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706904    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706906    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706909    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706911    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:06:02.710658 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.706914    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.706919    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707027    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707032    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707035    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707037    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707040    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707043    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707045    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707048    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707051    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707053    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707056    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707058    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707062    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:06:02.711110 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707064    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707067    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707069    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707072    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707075    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707078    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707080    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707083    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707085    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707088    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707090    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707093    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707095    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707098    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707101    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707104    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707106    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707109    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707111    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707114    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:06:02.711474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707116    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707119    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707121    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707124    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707126    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707129    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707131    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707134    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707137    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707139    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707142    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707144    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707147    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707150    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707152    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707156    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707160    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707163    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707166    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707169    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:06:02.712088 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707172    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707174    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707177    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707180    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707182    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707185    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707188    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707190    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707193    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707195    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707198    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707200    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707203    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707205    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707209    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707213    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707216    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707219    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707221    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:06:02.712560 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707224    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707227    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707230    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707232    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707236    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707238    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707241    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707244    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707247    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707250    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707252    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707256    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707259    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707262    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707265    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707267    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707270    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707273    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707275    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:06:02.713039 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707278    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:06:02.713497 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707280    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:06:02.713497 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707283    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:06:02.713497 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707285    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:06:02.713497 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707288    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:06:02.713497 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707291    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:06:02.713497 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707293    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:06:02.713497 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:02.707295    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:06:02.713497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.707301    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 21:06:02.713497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.707390    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 21:06:02.713497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.710211    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 21:06:02.713497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.711115    2579 server.go:1019] "Starting client certificate rotation"
Sep 18 21:06:02.713497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.711208    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 21:06:02.713497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.711241    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 21:06:02.713858 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.713660    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 21:06:02.717155 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.717140    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 21:06:02.726021 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.726005    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 18 21:06:02.727511 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.727492    2579 log.go:25] "Validated CRI v1 image API"
Sep 18 21:06:02.728342 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.728326    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 21:06:02.731188 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.731165    2579 fs.go:135] Filesystem UUIDs: map[2e57d77c-2e49-4a35-a1cd-073530d61e51:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f15563f5-8c95-48f0-aab3-f960ce133dac:/dev/nvme0n1p4]
Sep 18 21:06:02.731237 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.731188    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 21:06:02.736688 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.736570    2579 manager.go:216] Machine: {Timestamp:2026-09-18 21:06:02.735078962 +0000 UTC m=+0.342138516 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096256 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b051989532b2d7e093eb23990cfe3 SystemUUID:ec2b0519-8953-2b2d-7e09-3eb23990cfe3 BootID:e729b33d-6ed4-44d4-a192-79325f195a47 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dc:95:58:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dc:95:58:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:b6:af:fc:18:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 21:06:02.736688 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.736687    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 21:06:02.736793 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.736753    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 21:06:02.740662 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.740620    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 21:06:02.740789 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.740666    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-92.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 21:06:02.741449 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.741438    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 21:06:02.741484 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.741450    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 21:06:02.741484 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.741458    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 21:06:02.742825 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.742814    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 21:06:02.742887 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.742878    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 21:06:02.746019 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.746007    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 21:06:02.746132 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.746122    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 21:06:02.746175 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.746153    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 21:06:02.746175 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.746161    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 21:06:02.746175 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.746174    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 21:06:02.746286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.746182    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 18 21:06:02.746286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.746191    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 21:06:02.746497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.746487    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 21:06:02.746536 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.746505    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 21:06:02.749156 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.749139    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 21:06:02.751518 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.751505    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 21:06:02.751562 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.751536    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 21:06:02.753542 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.753527    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 21:06:02.753608 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.753549    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 21:06:02.753608 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.753558    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 21:06:02.753608 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.753568    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 21:06:02.753608 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.753589    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 21:06:02.753608 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.753599    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 21:06:02.753763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.753619    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 21:06:02.753763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.753629    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 21:06:02.753763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.753639    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 21:06:02.753763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.753664    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 21:06:02.753763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.753676    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 21:06:02.753763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.753686    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 21:06:02.754800 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.754779    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 21:06:02.755300 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.755288    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 21:06:02.755334 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.755305    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 21:06:02.758062 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.758039    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 21:06:02.758258 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.758236    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 21:06:02.758326 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.758306    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-92.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 21:06:02.759069 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.759058    2579 server.go:1268] "Started kubelet"
Sep 18 21:06:02.759153 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.759131    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 21:06:02.759206 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.759131    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 21:06:02.759250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.759224    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 21:06:02.759721 ip-10-0-130-92 systemd[1]: Started Kubernetes Kubelet.
Sep 18 21:06:02.760610 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.760547    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 21:06:02.761589 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.761565    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 21:06:02.764222 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.764201    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 21:06:02.767581 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.767560    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 21:06:02.769401 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.769384    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 21:06:02.769526 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.769506    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-92.ec2.internal\" not found"
Sep 18 21:06:02.770011 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.769994    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 21:06:02.770011 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.769998    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 21:06:02.770143 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.770036    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 21:06:02.770143 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.770085    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 21:06:02.770143 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.770093    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 21:06:02.770273 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.770165    2579 factory.go:55] Registering systemd factory
Sep 18 21:06:02.770273 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.770181    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 18 21:06:02.770392 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.770378    2579 factory.go:153] Registering CRI-O factory
Sep 18 21:06:02.770451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.770393    2579 factory.go:223] Registration of the crio container factory successfully
Sep 18 21:06:02.770502 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.770458    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 21:06:02.770502 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.770475    2579 factory.go:103] Registering Raw factory
Sep 18 21:06:02.770502 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.770488    2579 manager.go:1196] Started watching for new ooms in manager
Sep 18 21:06:02.770935 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.770918    2579 manager.go:349] Starting recovery of all containers
Sep 18 21:06:02.777425 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.777401    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 21:06:02.777563 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.777538    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 21:06:02.778858 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.777624    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-92.ec2.internal.18d6864bf77aabbf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-92.ec2.internal,UID:ip-10-0-130-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-92.ec2.internal,},FirstTimestamp:2026-09-18 21:06:02.759031743 +0000 UTC m=+0.366091293,LastTimestamp:2026-09-18 21:06:02.759031743 +0000 UTC m=+0.366091293,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-92.ec2.internal,}"
Sep 18 21:06:02.783668 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.783637    2579 manager.go:354] Recovery completed
Sep 18 21:06:02.788688 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.788636    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:06:02.790228 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.790212    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:06:02.790301 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.790241    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:06:02.790301 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.790254    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:06:02.790789 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.790777    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 21:06:02.790789 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.790783    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:06:02.790902 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.790787    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 21:06:02.790902 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.790819    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 21:06:02.791424 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.791411    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:06:02.791474 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.791434    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:06:02.791474 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.791446    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:06:02.792294 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.792283    2579 policy_none.go:49] "None policy: Start"
Sep 18 21:06:02.792350 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.792298    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 21:06:02.792350 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.792309    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 21:06:02.792717 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.792637    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-92.ec2.internal.18d6864bf956ab34  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-92.ec2.internal,UID:ip-10-0-130-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-92.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-92.ec2.internal,},FirstTimestamp:2026-09-18 21:06:02.79022674 +0000 UTC m=+0.397286289,LastTimestamp:2026-09-18 21:06:02.79022674 +0000 UTC m=+0.397286289,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-92.ec2.internal,}"
Sep 18 21:06:02.793423 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.793410    2579 policy_none.go:47] "Start"
Sep 18 21:06:02.801842 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.801771    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-92.ec2.internal.18d6864bf956fd26  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-92.ec2.internal,UID:ip-10-0-130-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-92.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-92.ec2.internal,},FirstTimestamp:2026-09-18 21:06:02.790247718 +0000 UTC m=+0.397307268,LastTimestamp:2026-09-18 21:06:02.790247718 +0000 UTC m=+0.397307268,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-92.ec2.internal,}"
Sep 18 21:06:02.808091 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.808021    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-92.ec2.internal.18d6864bf9572652  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-92.ec2.internal,UID:ip-10-0-130-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-92.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-92.ec2.internal,},FirstTimestamp:2026-09-18 21:06:02.790258258 +0000 UTC m=+0.397317806,LastTimestamp:2026-09-18 21:06:02.790258258 +0000 UTC m=+0.397317806,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-92.ec2.internal,}"
Sep 18 21:06:02.812812 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.812743    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-92.ec2.internal.18d6864bf956ab34\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-92.ec2.internal.18d6864bf956ab34  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-92.ec2.internal,UID:ip-10-0-130-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-92.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-92.ec2.internal,},FirstTimestamp:2026-09-18 21:06:02.79022674 +0000 UTC m=+0.397286289,LastTimestamp:2026-09-18 21:06:02.791423685 +0000 UTC m=+0.398483233,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-92.ec2.internal,}"
Sep 18 21:06:02.819158 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.819100    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-92.ec2.internal.18d6864bf956fd26\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-92.ec2.internal.18d6864bf956fd26  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-92.ec2.internal,UID:ip-10-0-130-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-92.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-92.ec2.internal,},FirstTimestamp:2026-09-18 21:06:02.790247718 +0000 UTC m=+0.397307268,LastTimestamp:2026-09-18 21:06:02.791438983 +0000 UTC m=+0.398498532,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-92.ec2.internal,}"
Sep 18 21:06:02.841636 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.821934    2579 manager.go:337] "Starting Device Plugin manager"
Sep 18 21:06:02.841636 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.821975    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 21:06:02.841636 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.821987    2579 server.go:87] "Starting device plugin registration server"
Sep 18 21:06:02.841636 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.822201    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 21:06:02.841636 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.822212    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 21:06:02.841636 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.822305    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 21:06:02.841636 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.822378    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 21:06:02.841636 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.822386    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 21:06:02.841636 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.823848    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 21:06:02.841636 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.823902    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-92.ec2.internal\" not found"
Sep 18 21:06:02.841636 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.824497    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-92.ec2.internal.18d6864bf9572652\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-92.ec2.internal.18d6864bf9572652  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-92.ec2.internal,UID:ip-10-0-130-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-92.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-92.ec2.internal,},FirstTimestamp:2026-09-18 21:06:02.790258258 +0000 UTC m=+0.397317806,LastTimestamp:2026-09-18 21:06:02.791452405 +0000 UTC m=+0.398511953,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-92.ec2.internal,}"
Sep 18 21:06:02.841636 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.836505    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-46b5r"
Sep 18 21:06:02.841636 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.839029    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-92.ec2.internal.18d6864bfc204f5e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-92.ec2.internal,UID:ip-10-0-130-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-130-92.ec2.internal,},FirstTimestamp:2026-09-18 21:06:02.836995934 +0000 UTC m=+0.444055473,LastTimestamp:2026-09-18 21:06:02.836995934 +0000 UTC m=+0.444055473,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-92.ec2.internal,}"
Sep 18 21:06:02.844841 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.844821    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-46b5r"
Sep 18 21:06:02.914063 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.914039    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 21:06:02.915111 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.915096    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 21:06:02.915164 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.915137    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 21:06:02.915164 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.915157    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 21:06:02.915254 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.915190    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 21:06:02.922874 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.922855    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:06:02.924168 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.924149    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:06:02.924240 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.924181    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:06:02.924240 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.924196    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:06:02.924240 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.924221    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-92.ec2.internal"
Sep 18 21:06:02.926835 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.926820    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:06:02.936501 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:02.936486    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-92.ec2.internal"
Sep 18 21:06:02.936586 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.936507    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-92.ec2.internal\": node \"ip-10-0-130-92.ec2.internal\" not found"
Sep 18 21:06:02.976995 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:02.976938    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-92.ec2.internal\" not found"
Sep 18 21:06:03.016009 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.015982    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal"]
Sep 18 21:06:03.016084 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.016042    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:06:03.017436 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.017415    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:06:03.017498 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.017440    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:06:03.017498 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.017449    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:06:03.018963 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.018950    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:06:03.019081 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.019067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.019137 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.019101    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:06:03.019602 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.019588    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:06:03.019692 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.019614    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:06:03.019692 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.019619    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:06:03.019692 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.019628    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:06:03.019692 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.019662    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:06:03.019692 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.019675    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:06:03.021220 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.021206    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.021294 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.021229    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:06:03.021923 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.021909    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:06:03.021993 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.021934    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:06:03.021993 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.021943    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:06:03.054258 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.054240    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-92.ec2.internal\" not found" node="ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.058602 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.058588    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-92.ec2.internal\" not found" node="ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.078017 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.077997    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-92.ec2.internal\" not found"
Sep 18 21:06:03.171369 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.171345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cbd45bdbcda509ce01b35a6f5d64d3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal\" (UID: \"0cbd45bdbcda509ce01b35a6f5d64d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.171456 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.171373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cbd45bdbcda509ce01b35a6f5d64d3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal\" (UID: \"0cbd45bdbcda509ce01b35a6f5d64d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.171456 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.171398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c6ac307dfcd56096fce875fa1e1ed1ca-config\") pod \"kube-apiserver-proxy-ip-10-0-130-92.ec2.internal\" (UID: \"c6ac307dfcd56096fce875fa1e1ed1ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.178902 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.178877    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-92.ec2.internal\" not found"
Sep 18 21:06:03.272257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.272200    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cbd45bdbcda509ce01b35a6f5d64d3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal\" (UID: \"0cbd45bdbcda509ce01b35a6f5d64d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.272257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.272234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0cbd45bdbcda509ce01b35a6f5d64d3d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal\" (UID: \"0cbd45bdbcda509ce01b35a6f5d64d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.272382 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.272276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cbd45bdbcda509ce01b35a6f5d64d3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal\" (UID: \"0cbd45bdbcda509ce01b35a6f5d64d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.272382 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.272294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c6ac307dfcd56096fce875fa1e1ed1ca-config\") pod \"kube-apiserver-proxy-ip-10-0-130-92.ec2.internal\" (UID: \"c6ac307dfcd56096fce875fa1e1ed1ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.272382 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.272324    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c6ac307dfcd56096fce875fa1e1ed1ca-config\") pod \"kube-apiserver-proxy-ip-10-0-130-92.ec2.internal\" (UID: \"c6ac307dfcd56096fce875fa1e1ed1ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.272382 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.272351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cbd45bdbcda509ce01b35a6f5d64d3d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal\" (UID: \"0cbd45bdbcda509ce01b35a6f5d64d3d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.279288 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.279268    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-92.ec2.internal\" not found"
Sep 18 21:06:03.358455 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.358437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.363373 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.363035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.380256 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.380229    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-92.ec2.internal\" not found"
Sep 18 21:06:03.480851 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.480814    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-92.ec2.internal\" not found"
Sep 18 21:06:03.581363 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.581300    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-92.ec2.internal\" not found"
Sep 18 21:06:03.632338 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.632310    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:06:03.670390 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.670363    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.678824 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.678803    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:06:03.707819 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.707792    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 21:06:03.709099 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.709082    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.711193 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.711175    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 21:06:03.711300 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.711280    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 21:06:03.711365 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.711288    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 21:06:03.711365 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.711309    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 21:06:03.711365 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.711319    2579 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a09043fc62cbf4739bb1111fa9d2d344-ac78f8b995c8d1a0.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.130.92:60306->32.196.69.98:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:03.747842 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.747819    2579 apiserver.go:52] "Watching apiserver"
Sep 18 21:06:03.767935 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.767906    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 21:06:03.768039 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.767936    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 21:06:03.768416 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.768383    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-rzd48","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr","openshift-cluster-node-tuning-operator/tuned-rcfh2","openshift-dns/node-resolver-khlzv","openshift-multus/multus-additional-cni-plugins-gcgn5","openshift-multus/network-metrics-daemon-bt5w7","kube-system/konnectivity-agent-27drt","openshift-image-registry/node-ca-tkw2z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal","openshift-multus/multus-45p77","openshift-network-diagnostics/network-check-target-pm2mb","openshift-network-operator/iptables-alerter-khjbr","openshift-ovn-kubernetes/ovnkube-node-9fpv5"]
Sep 18 21:06:03.770097 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.770071    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:03.770210 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.770183    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:03.771463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.771440    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.773022 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.773004    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.774423 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.774402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-khlzv"
Sep 18 21:06:03.776003 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.775985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.776813 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.776792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-socket-dir\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.776908 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.776820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-registration-dir\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.776908 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.776839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-device-dir\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.776908 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.776855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-etc-selinux\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.776908 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.776880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-kubelet-config\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:03.776908 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.776903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-dbus\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:03.777106 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.776946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-sys-fs\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.777106 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.776996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znq4s\" (UniqueName: \"kubernetes.io/projected/74da1b67-8d01-4ef9-9de1-7349d34cc6de-kube-api-access-znq4s\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.777106 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.777020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:03.777106 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.777038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.777336 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.777322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:03.777405 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.777388    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:03.778592 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.778572    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fstlz\""
Sep 18 21:06:03.778688 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.778669    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-27drt"
Sep 18 21:06:03.778774 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.778685    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 21:06:03.778774 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.778686    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 21:06:03.779629 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.779611    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 21:06:03.780036 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.780019    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 21:06:03.780154 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.780135    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jnwtj\""
Sep 18 21:06:03.780214 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.780168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tkw2z"
Sep 18 21:06:03.781760 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.781739    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.783201 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.783185    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:03.783270 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.783246    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:03.784559 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.784541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-khjbr"
Sep 18 21:06:03.785928 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.785913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.796358 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.796333    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mc7q5\""
Sep 18 21:06:03.796471 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.796402    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 21:06:03.796695 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.796672    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 21:06:03.796760 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.796697    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 21:06:03.847386 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.847245    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c7cw4\""
Sep 18 21:06:03.849660 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.849631    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 21:01:03 +0000 UTC" deadline="2028-06-29 10:37:14.637778994 +0000 UTC"
Sep 18 21:06:03.849660 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.849660    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15589h31m10.788123408s"
Sep 18 21:06:03.849800 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.849699    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 21:06:03.850131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.850113    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 21:06:03.850278 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.850262    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 21:06:03.851036 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.851016    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 21:06:03.851226 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.851206    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4x6fl\""
Sep 18 21:06:03.851711 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.851686    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 21:06:03.856245 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.856222    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 21:06:03.856353 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.856283    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 21:06:03.856353 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.856275    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:06:03.856479 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.856367    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 21:06:03.858168 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.858147    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 21:06:03.858168 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.858160    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 21:06:03.858168 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.858167    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 21:06:03.858366 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.858165    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xmk5m\""
Sep 18 21:06:03.858366 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.858196    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rbkpq\""
Sep 18 21:06:03.862477 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.862457    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:06:03.862572 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.862499    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 21:06:03.866326 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.866308    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 21:06:03.866443 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.866426    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 21:06:03.866659 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.866631    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 21:06:03.867017 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.867002    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w72wr\""
Sep 18 21:06:03.870970 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.870947    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 21:06:03.877202 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-var-lib-cni-bin\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.877343 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-run-k8s-cni-cncf-io\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.877343 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-kubernetes\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.877343 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-lib-modules\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.877445 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abfd23c2-a5d2-442e-a53b-ba3556428037-system-cni-dir\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.877445 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-cni-netd\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.877445 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgpgt\" (UniqueName: \"kubernetes.io/projected/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-kube-api-access-vgpgt\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:03.877445 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc1f208a-bfe4-49d1-9dac-1214ace566d8-host\") pod \"node-ca-tkw2z\" (UID: \"dc1f208a-bfe4-49d1-9dac-1214ace566d8\") " pod="openshift-image-registry/node-ca-tkw2z"
Sep 18 21:06:03.877626 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-multus-cni-dir\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.877626 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fada2848-646b-456e-bd95-606b2da5c298-cni-binary-copy\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.877626 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-etc-kubernetes\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.877626 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-ovnkube-script-lib\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.877626 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-socket-dir\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.877626 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-registration-dir\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.877626 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-etc-selinux\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-registration-dir\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtbhq\" (UniqueName: \"kubernetes.io/projected/b57c9901-f383-4e37-b782-dfcfd75ccbc4-kube-api-access-jtbhq\") pod \"iptables-alerter-khjbr\" (UID: \"b57c9901-f383-4e37-b782-dfcfd75ccbc4\") " pod="openshift-network-operator/iptables-alerter-khjbr"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-socket-dir\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-ovn-node-metrics-cert\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-etc-selinux\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx5fg\" (UniqueName: \"kubernetes.io/projected/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-kube-api-access-rx5fg\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-sysconfig\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877813    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9187f979-a13d-4dc5-9fa2-362a99aaaee4-tmp\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-ovnkube-config\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b57c9901-f383-4e37-b782-dfcfd75ccbc4-iptables-alerter-script\") pod \"iptables-alerter-khjbr\" (UID: \"b57c9901-f383-4e37-b782-dfcfd75ccbc4\") " pod="openshift-network-operator/iptables-alerter-khjbr"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-run-openvswitch\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63bb4238-7a5e-462c-9b50-45e3b2813957-konnectivity-ca\") pod \"konnectivity-agent-27drt\" (UID: \"63bb4238-7a5e-462c-9b50-45e3b2813957\") " pod="kube-system/konnectivity-agent-27drt"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-run-netns\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.877977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.877975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cbaf64-d066-42f3-a262-8e56d161434a-tmp-dir\") pod \"node-resolver-khlzv\" (UID: \"13cbaf64-d066-42f3-a262-8e56d161434a\") " pod="openshift-dns/node-resolver-khlzv"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abfd23c2-a5d2-442e-a53b-ba3556428037-cnibin\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abfd23c2-a5d2-442e-a53b-ba3556428037-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-multus-socket-dir-parent\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mss75\" (UniqueName: \"kubernetes.io/projected/fada2848-646b-456e-bd95-606b2da5c298-kube-api-access-mss75\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-env-overrides\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-sysctl-conf\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-slash\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-log-socket\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-node-log\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-run-ovn-kubernetes\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-device-dir\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.878380    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-device-dir\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abfd23c2-a5d2-442e-a53b-ba3556428037-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.878763 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.878485    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret podName:5b2108fb-5c03-49d2-a864-2ef38a1fbaab nodeName:}" failed. No retries permitted until 2026-09-18 21:06:04.378448677 +0000 UTC m=+1.985508222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret") pod "global-pull-secret-syncer-rzd48" (UID: "5b2108fb-5c03-49d2-a864-2ef38a1fbaab") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-var-lib-cni-multus\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b57c9901-f383-4e37-b782-dfcfd75ccbc4-host-slash\") pod \"iptables-alerter-khjbr\" (UID: \"b57c9901-f383-4e37-b782-dfcfd75ccbc4\") " pod="openshift-network-operator/iptables-alerter-khjbr"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-var-lib-openvswitch\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-host\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-sys-fs\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwxlp\" (UniqueName: \"kubernetes.io/projected/dc1f208a-bfe4-49d1-9dac-1214ace566d8-kube-api-access-lwxlp\") pod \"node-ca-tkw2z\" (UID: \"dc1f208a-bfe4-49d1-9dac-1214ace566d8\") " pod="openshift-image-registry/node-ca-tkw2z"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-run-multus-certs\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmngj\" (UniqueName: \"kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj\") pod \"network-check-target-pm2mb\" (UID: \"085474f0-4cdb-4b45-9fd2-f00a4e10af1c\") " pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbz78\" (UniqueName: \"kubernetes.io/projected/9187f979-a13d-4dc5-9fa2-362a99aaaee4-kube-api-access-sbz78\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878847    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/74da1b67-8d01-4ef9-9de1-7349d34cc6de-sys-fs\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24gck\" (UniqueName: \"kubernetes.io/projected/abfd23c2-a5d2-442e-a53b-ba3556428037-kube-api-access-24gck\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-kubelet-config\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:03.879308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-hostroot\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fada2848-646b-456e-bd95-606b2da5c298-multus-daemon-config\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-cni-bin\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878974    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-sysctl-d\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.878954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-kubelet-config\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc1f208a-bfe4-49d1-9dac-1214ace566d8-serviceca\") pod \"node-ca-tkw2z\" (UID: \"dc1f208a-bfe4-49d1-9dac-1214ace566d8\") " pod="openshift-image-registry/node-ca-tkw2z"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-run-netns\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-multus-conf-dir\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-tuned\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-dbus\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znq4s\" (UniqueName: \"kubernetes.io/projected/74da1b67-8d01-4ef9-9de1-7349d34cc6de-kube-api-access-znq4s\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-system-cni-dir\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-cnibin\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-run-systemd\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-etc-openvswitch\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879269    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-dbus\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:03.879931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-run-ovn\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.880441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-systemd\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.880441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-var-lib-kubelet\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.880441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63bb4238-7a5e-462c-9b50-45e3b2813957-agent-certs\") pod \"konnectivity-agent-27drt\" (UID: \"63bb4238-7a5e-462c-9b50-45e3b2813957\") " pod="kube-system/konnectivity-agent-27drt"
Sep 18 21:06:03.880441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-kubelet\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.880441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-modprobe-d\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.880441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-sys\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.880441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-var-lib-kubelet\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.880441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abfd23c2-a5d2-442e-a53b-ba3556428037-cni-binary-copy\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.880441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-systemd-units\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.880441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-run\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.880441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13cbaf64-d066-42f3-a262-8e56d161434a-hosts-file\") pod \"node-resolver-khlzv\" (UID: \"13cbaf64-d066-42f3-a262-8e56d161434a\") " pod="openshift-dns/node-resolver-khlzv"
Sep 18 21:06:03.880441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6wb7\" (UniqueName: \"kubernetes.io/projected/13cbaf64-d066-42f3-a262-8e56d161434a-kube-api-access-w6wb7\") pod \"node-resolver-khlzv\" (UID: \"13cbaf64-d066-42f3-a262-8e56d161434a\") " pod="openshift-dns/node-resolver-khlzv"
Sep 18 21:06:03.880441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.879584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abfd23c2-a5d2-442e-a53b-ba3556428037-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.893709 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.893669    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 21:06:03.896888 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.896868    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 21:06:03.899006 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.898989    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 21:06:03.900363 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.900346    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 21:06:03.903881 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.903864    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znq4s\" (UniqueName: \"kubernetes.io/projected/74da1b67-8d01-4ef9-9de1-7349d34cc6de-kube-api-access-znq4s\") pod \"aws-ebs-csi-driver-node-pc7mr\" (UID: \"74da1b67-8d01-4ef9-9de1-7349d34cc6de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:03.915395 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.915371    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 21:06:03.917698 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.917659    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mkx42\""
Sep 18 21:06:03.956708 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.956683    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kkc9x"
Sep 18 21:06:03.976486 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.976467    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kkc9x"
Sep 18 21:06:03.979803 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.979783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-multus-socket-dir-parent\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.979914 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.979809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mss75\" (UniqueName: \"kubernetes.io/projected/fada2848-646b-456e-bd95-606b2da5c298-kube-api-access-mss75\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.979914 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.979826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-env-overrides\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.979914 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.979844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-sysctl-conf\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.979914 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.979858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-slash\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.979940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-multus-socket-dir-parent\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.980130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.979951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-sysctl-conf\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.980130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-slash\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-log-socket\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-node-log\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-run-ovn-kubernetes\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-log-socket\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abfd23c2-a5d2-442e-a53b-ba3556428037-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.980426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-run-ovn-kubernetes\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-node-log\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-var-lib-cni-multus\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.980426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-env-overrides\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b57c9901-f383-4e37-b782-dfcfd75ccbc4-host-slash\") pod \"iptables-alerter-khjbr\" (UID: \"b57c9901-f383-4e37-b782-dfcfd75ccbc4\") " pod="openshift-network-operator/iptables-alerter-khjbr"
Sep 18 21:06:03.980426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980289    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-var-lib-cni-multus\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.980426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b57c9901-f383-4e37-b782-dfcfd75ccbc4-host-slash\") pod \"iptables-alerter-khjbr\" (UID: \"b57c9901-f383-4e37-b782-dfcfd75ccbc4\") " pod="openshift-network-operator/iptables-alerter-khjbr"
Sep 18 21:06:03.980426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-var-lib-openvswitch\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-var-lib-openvswitch\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-host\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.980426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:03.980426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwxlp\" (UniqueName: \"kubernetes.io/projected/dc1f208a-bfe4-49d1-9dac-1214ace566d8-kube-api-access-lwxlp\") pod \"node-ca-tkw2z\" (UID: \"dc1f208a-bfe4-49d1-9dac-1214ace566d8\") " pod="openshift-image-registry/node-ca-tkw2z"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-run-multus-certs\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-host\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmngj\" (UniqueName: \"kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj\") pod \"network-check-target-pm2mb\" (UID: \"085474f0-4cdb-4b45-9fd2-f00a4e10af1c\") " pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbz78\" (UniqueName: \"kubernetes.io/projected/9187f979-a13d-4dc5-9fa2-362a99aaaee4-kube-api-access-sbz78\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.980532    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:03.980579    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs podName:0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa nodeName:}" failed. No retries permitted until 2026-09-18 21:06:04.480565943 +0000 UTC m=+2.087625482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs") pod "network-metrics-daemon-bt5w7" (UID: "0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-run-multus-certs\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24gck\" (UniqueName: \"kubernetes.io/projected/abfd23c2-a5d2-442e-a53b-ba3556428037-kube-api-access-24gck\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-hostroot\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fada2848-646b-456e-bd95-606b2da5c298-multus-daemon-config\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-cni-bin\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980701    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-hostroot\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-sysctl-d\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/abfd23c2-a5d2-442e-a53b-ba3556428037-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-cni-bin\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.980949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc1f208a-bfe4-49d1-9dac-1214ace566d8-serviceca\") pod \"node-ca-tkw2z\" (UID: \"dc1f208a-bfe4-49d1-9dac-1214ace566d8\") " pod="openshift-image-registry/node-ca-tkw2z"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-run-netns\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-run-netns\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-sysctl-d\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-multus-conf-dir\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-tuned\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980894    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-system-cni-dir\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-cnibin\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-multus-conf-dir\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-run-systemd\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-run-systemd\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-etc-openvswitch\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-run-ovn\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-cnibin\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-systemd\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.980991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-etc-openvswitch\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981029    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-system-cni-dir\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.981755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-run-ovn\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-var-lib-kubelet\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63bb4238-7a5e-462c-9b50-45e3b2813957-agent-certs\") pod \"konnectivity-agent-27drt\" (UID: \"63bb4238-7a5e-462c-9b50-45e3b2813957\") " pod="kube-system/konnectivity-agent-27drt"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-var-lib-kubelet\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-kubelet\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-modprobe-d\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-kubelet\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-systemd\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-sys\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-var-lib-kubelet\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc1f208a-bfe4-49d1-9dac-1214ace566d8-serviceca\") pod \"node-ca-tkw2z\" (UID: \"dc1f208a-bfe4-49d1-9dac-1214ace566d8\") " pod="openshift-image-registry/node-ca-tkw2z"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abfd23c2-a5d2-442e-a53b-ba3556428037-cni-binary-copy\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fada2848-646b-456e-bd95-606b2da5c298-multus-daemon-config\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-systemd-units\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981231    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-modprobe-d\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-run\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981275    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-var-lib-kubelet\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-sys\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.982565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-systemd-units\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-run\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13cbaf64-d066-42f3-a262-8e56d161434a-hosts-file\") pod \"node-resolver-khlzv\" (UID: \"13cbaf64-d066-42f3-a262-8e56d161434a\") " pod="openshift-dns/node-resolver-khlzv"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6wb7\" (UniqueName: \"kubernetes.io/projected/13cbaf64-d066-42f3-a262-8e56d161434a-kube-api-access-w6wb7\") pod \"node-resolver-khlzv\" (UID: \"13cbaf64-d066-42f3-a262-8e56d161434a\") " pod="openshift-dns/node-resolver-khlzv"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abfd23c2-a5d2-442e-a53b-ba3556428037-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-var-lib-cni-bin\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981604    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13cbaf64-d066-42f3-a262-8e56d161434a-hosts-file\") pod \"node-resolver-khlzv\" (UID: \"13cbaf64-d066-42f3-a262-8e56d161434a\") " pod="openshift-dns/node-resolver-khlzv"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-run-k8s-cni-cncf-io\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-kubernetes\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-lib-modules\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981711    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-run-k8s-cni-cncf-io\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-host-var-lib-cni-bin\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abfd23c2-a5d2-442e-a53b-ba3556428037-system-cni-dir\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-cni-netd\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/abfd23c2-a5d2-442e-a53b-ba3556428037-system-cni-dir\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-kubernetes\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgpgt\" (UniqueName: \"kubernetes.io/projected/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-kube-api-access-vgpgt\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:03.983521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-lib-modules\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc1f208a-bfe4-49d1-9dac-1214ace566d8-host\") pod \"node-ca-tkw2z\" (UID: \"dc1f208a-bfe4-49d1-9dac-1214ace566d8\") " pod="openshift-image-registry/node-ca-tkw2z"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981829    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-cni-netd\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981829    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/abfd23c2-a5d2-442e-a53b-ba3556428037-cni-binary-copy\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981878    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc1f208a-bfe4-49d1-9dac-1214ace566d8-host\") pod \"node-ca-tkw2z\" (UID: \"dc1f208a-bfe4-49d1-9dac-1214ace566d8\") " pod="openshift-image-registry/node-ca-tkw2z"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/abfd23c2-a5d2-442e-a53b-ba3556428037-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.981959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-multus-cni-dir\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fada2848-646b-456e-bd95-606b2da5c298-cni-binary-copy\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-etc-kubernetes\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-multus-cni-dir\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-ovnkube-script-lib\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982077    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fada2848-646b-456e-bd95-606b2da5c298-etc-kubernetes\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtbhq\" (UniqueName: \"kubernetes.io/projected/b57c9901-f383-4e37-b782-dfcfd75ccbc4-kube-api-access-jtbhq\") pod \"iptables-alerter-khjbr\" (UID: \"b57c9901-f383-4e37-b782-dfcfd75ccbc4\") " pod="openshift-network-operator/iptables-alerter-khjbr"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-ovn-node-metrics-cert\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx5fg\" (UniqueName: \"kubernetes.io/projected/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-kube-api-access-rx5fg\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-sysconfig\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9187f979-a13d-4dc5-9fa2-362a99aaaee4-tmp\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-ovnkube-config\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.984451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b57c9901-f383-4e37-b782-dfcfd75ccbc4-iptables-alerter-script\") pod \"iptables-alerter-khjbr\" (UID: \"b57c9901-f383-4e37-b782-dfcfd75ccbc4\") " pod="openshift-network-operator/iptables-alerter-khjbr"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-run-openvswitch\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63bb4238-7a5e-462c-9b50-45e3b2813957-konnectivity-ca\") pod \"konnectivity-agent-27drt\" (UID: \"63bb4238-7a5e-462c-9b50-45e3b2813957\") " pod="kube-system/konnectivity-agent-27drt"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-run-netns\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cbaf64-d066-42f3-a262-8e56d161434a-tmp-dir\") pod \"node-resolver-khlzv\" (UID: \"13cbaf64-d066-42f3-a262-8e56d161434a\") " pod="openshift-dns/node-resolver-khlzv"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abfd23c2-a5d2-442e-a53b-ba3556428037-cnibin\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abfd23c2-a5d2-442e-a53b-ba3556428037-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fada2848-646b-456e-bd95-606b2da5c298-cni-binary-copy\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/abfd23c2-a5d2-442e-a53b-ba3556428037-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-host-run-netns\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-ovnkube-script-lib\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/abfd23c2-a5d2-442e-a53b-ba3556428037-cnibin\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.982799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-sysconfig\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.983001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13cbaf64-d066-42f3-a262-8e56d161434a-tmp-dir\") pod \"node-resolver-khlzv\" (UID: \"13cbaf64-d066-42f3-a262-8e56d161434a\") " pod="openshift-dns/node-resolver-khlzv"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.983022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-run-openvswitch\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.983139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63bb4238-7a5e-462c-9b50-45e3b2813957-konnectivity-ca\") pod \"konnectivity-agent-27drt\" (UID: \"63bb4238-7a5e-462c-9b50-45e3b2813957\") " pod="kube-system/konnectivity-agent-27drt"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.983326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b57c9901-f383-4e37-b782-dfcfd75ccbc4-iptables-alerter-script\") pod \"iptables-alerter-khjbr\" (UID: \"b57c9901-f383-4e37-b782-dfcfd75ccbc4\") " pod="openshift-network-operator/iptables-alerter-khjbr"
Sep 18 21:06:03.985257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.983365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-ovnkube-config\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.985821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.983567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9187f979-a13d-4dc5-9fa2-362a99aaaee4-etc-tuned\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.985821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.984916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-ovn-node-metrics-cert\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:03.985821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.985209    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9187f979-a13d-4dc5-9fa2-362a99aaaee4-tmp\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:03.985821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:03.985278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63bb4238-7a5e-462c-9b50-45e3b2813957-agent-certs\") pod \"konnectivity-agent-27drt\" (UID: \"63bb4238-7a5e-462c-9b50-45e3b2813957\") " pod="kube-system/konnectivity-agent-27drt"
Sep 18 21:06:04.001424 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:04.001400    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:06:04.001424 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:04.001428    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:06:04.001598 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:04.001443    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mmngj for pod openshift-network-diagnostics/network-check-target-pm2mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:04.001598 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:04.001512    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj podName:085474f0-4cdb-4b45-9fd2-f00a4e10af1c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:04.501493482 +0000 UTC m=+2.108553026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mmngj" (UniqueName: "kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj") pod "network-check-target-pm2mb" (UID: "085474f0-4cdb-4b45-9fd2-f00a4e10af1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:04.002847 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.002821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24gck\" (UniqueName: \"kubernetes.io/projected/abfd23c2-a5d2-442e-a53b-ba3556428037-kube-api-access-24gck\") pod \"multus-additional-cni-plugins-gcgn5\" (UID: \"abfd23c2-a5d2-442e-a53b-ba3556428037\") " pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:04.002949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.002827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwxlp\" (UniqueName: \"kubernetes.io/projected/dc1f208a-bfe4-49d1-9dac-1214ace566d8-kube-api-access-lwxlp\") pod \"node-ca-tkw2z\" (UID: \"dc1f208a-bfe4-49d1-9dac-1214ace566d8\") " pod="openshift-image-registry/node-ca-tkw2z"
Sep 18 21:06:04.002949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.002871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgpgt\" (UniqueName: \"kubernetes.io/projected/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-kube-api-access-vgpgt\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:04.003273 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.003243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtbhq\" (UniqueName: \"kubernetes.io/projected/b57c9901-f383-4e37-b782-dfcfd75ccbc4-kube-api-access-jtbhq\") pod \"iptables-alerter-khjbr\" (UID: \"b57c9901-f383-4e37-b782-dfcfd75ccbc4\") " pod="openshift-network-operator/iptables-alerter-khjbr"
Sep 18 21:06:04.004246 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.004221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbz78\" (UniqueName: \"kubernetes.io/projected/9187f979-a13d-4dc5-9fa2-362a99aaaee4-kube-api-access-sbz78\") pod \"tuned-rcfh2\" (UID: \"9187f979-a13d-4dc5-9fa2-362a99aaaee4\") " pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:04.004446 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.004427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6wb7\" (UniqueName: \"kubernetes.io/projected/13cbaf64-d066-42f3-a262-8e56d161434a-kube-api-access-w6wb7\") pod \"node-resolver-khlzv\" (UID: \"13cbaf64-d066-42f3-a262-8e56d161434a\") " pod="openshift-dns/node-resolver-khlzv"
Sep 18 21:06:04.004809 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.004791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx5fg\" (UniqueName: \"kubernetes.io/projected/9dda4277-1b86-4bad-8334-d9a06d7d9b3b-kube-api-access-rx5fg\") pod \"ovnkube-node-9fpv5\" (UID: \"9dda4277-1b86-4bad-8334-d9a06d7d9b3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:04.005995 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.005974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mss75\" (UniqueName: \"kubernetes.io/projected/fada2848-646b-456e-bd95-606b2da5c298-kube-api-access-mss75\") pod \"multus-45p77\" (UID: \"fada2848-646b-456e-bd95-606b2da5c298\") " pod="openshift-multus/multus-45p77"
Sep 18 21:06:04.012489 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:04.012461    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbd45bdbcda509ce01b35a6f5d64d3d.slice/crio-680704cdd1bfd4c88139e17183bf5ee0726a1697fb41faf2e77bd9480fae8697 WatchSource:0}: Error finding container 680704cdd1bfd4c88139e17183bf5ee0726a1697fb41faf2e77bd9480fae8697: Status 404 returned error can't find the container with id 680704cdd1bfd4c88139e17183bf5ee0726a1697fb41faf2e77bd9480fae8697
Sep 18 21:06:04.013599 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:04.013575    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ac307dfcd56096fce875fa1e1ed1ca.slice/crio-0d2cb6b2810586b0d39017ac1ac28cae66b8a6e1815dc681cc6eae3f15c3ee82 WatchSource:0}: Error finding container 0d2cb6b2810586b0d39017ac1ac28cae66b8a6e1815dc681cc6eae3f15c3ee82: Status 404 returned error can't find the container with id 0d2cb6b2810586b0d39017ac1ac28cae66b8a6e1815dc681cc6eae3f15c3ee82
Sep 18 21:06:04.019059 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.019036    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:06:04.108382 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.108334    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr"
Sep 18 21:06:04.116221 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:04.116201    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74da1b67_8d01_4ef9_9de1_7349d34cc6de.slice/crio-e3c9193e4403c1bce11fc3fec3db5de0b0252d59d44b25d810a1b03ef482eaa6 WatchSource:0}: Error finding container e3c9193e4403c1bce11fc3fec3db5de0b0252d59d44b25d810a1b03ef482eaa6: Status 404 returned error can't find the container with id e3c9193e4403c1bce11fc3fec3db5de0b0252d59d44b25d810a1b03ef482eaa6
Sep 18 21:06:04.127597 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.127581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rcfh2"
Sep 18 21:06:04.134659 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:04.134640    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9187f979_a13d_4dc5_9fa2_362a99aaaee4.slice/crio-fcd95dd59419c7df7c42dd034d58c6d7e6e0fa1e08f67b132cc3dd56bebe59b0 WatchSource:0}: Error finding container fcd95dd59419c7df7c42dd034d58c6d7e6e0fa1e08f67b132cc3dd56bebe59b0: Status 404 returned error can't find the container with id fcd95dd59419c7df7c42dd034d58c6d7e6e0fa1e08f67b132cc3dd56bebe59b0
Sep 18 21:06:04.137650 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.137570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-khlzv"
Sep 18 21:06:04.144463 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:04.144442    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13cbaf64_d066_42f3_a262_8e56d161434a.slice/crio-e7cf24ae0dcd786253fde31fd9dd20a5267f707325c3d0ee570c838ebeeff996 WatchSource:0}: Error finding container e7cf24ae0dcd786253fde31fd9dd20a5267f707325c3d0ee570c838ebeeff996: Status 404 returned error can't find the container with id e7cf24ae0dcd786253fde31fd9dd20a5267f707325c3d0ee570c838ebeeff996
Sep 18 21:06:04.161489 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.161471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gcgn5"
Sep 18 21:06:04.161793 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.161776    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:06:04.168272 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:04.168255    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfd23c2_a5d2_442e_a53b_ba3556428037.slice/crio-843b1e168b597f7f632fb054f8d08c8649daf94336c4f64dcddce56e11de5f76 WatchSource:0}: Error finding container 843b1e168b597f7f632fb054f8d08c8649daf94336c4f64dcddce56e11de5f76: Status 404 returned error can't find the container with id 843b1e168b597f7f632fb054f8d08c8649daf94336c4f64dcddce56e11de5f76
Sep 18 21:06:04.175408 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.175393    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-27drt"
Sep 18 21:06:04.182114 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:04.182093    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bb4238_7a5e_462c_9b50_45e3b2813957.slice/crio-f6aa9c6e5e0e7f84d8b9a08d3ebb8a801af9ae796917af7b1042c421aaaa34c6 WatchSource:0}: Error finding container f6aa9c6e5e0e7f84d8b9a08d3ebb8a801af9ae796917af7b1042c421aaaa34c6: Status 404 returned error can't find the container with id f6aa9c6e5e0e7f84d8b9a08d3ebb8a801af9ae796917af7b1042c421aaaa34c6
Sep 18 21:06:04.192542 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.192518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tkw2z"
Sep 18 21:06:04.199181 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:04.199164    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1f208a_bfe4_49d1_9dac_1214ace566d8.slice/crio-3de2d0075ffe3d1fc1595e901af20e8d923c8c8ec7d34900883c403aebeed6f1 WatchSource:0}: Error finding container 3de2d0075ffe3d1fc1595e901af20e8d923c8c8ec7d34900883c403aebeed6f1: Status 404 returned error can't find the container with id 3de2d0075ffe3d1fc1595e901af20e8d923c8c8ec7d34900883c403aebeed6f1
Sep 18 21:06:04.214446 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.214425    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-45p77"
Sep 18 21:06:04.221105 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:04.221079    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfada2848_646b_456e_bd95_606b2da5c298.slice/crio-ed78811bf15e367b5841f4b77dcb2c5593882cb6cd96935785899005ccb7f2de WatchSource:0}: Error finding container ed78811bf15e367b5841f4b77dcb2c5593882cb6cd96935785899005ccb7f2de: Status 404 returned error can't find the container with id ed78811bf15e367b5841f4b77dcb2c5593882cb6cd96935785899005ccb7f2de
Sep 18 21:06:04.227998 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.227982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-khjbr"
Sep 18 21:06:04.232388 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.232369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:04.235407 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:04.235388    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57c9901_f383_4e37_b782_dfcfd75ccbc4.slice/crio-124d652b8b2c163d7ae46dc42245d11754c1c549e2cc88db13ff17129279ef88 WatchSource:0}: Error finding container 124d652b8b2c163d7ae46dc42245d11754c1c549e2cc88db13ff17129279ef88: Status 404 returned error can't find the container with id 124d652b8b2c163d7ae46dc42245d11754c1c549e2cc88db13ff17129279ef88
Sep 18 21:06:04.239991 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:04.239972    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dda4277_1b86_4bad_8334_d9a06d7d9b3b.slice/crio-453b86544165dd080a05a8bedeeb7fc86dad59a3af705f6912ce1915c6922910 WatchSource:0}: Error finding container 453b86544165dd080a05a8bedeeb7fc86dad59a3af705f6912ce1915c6922910: Status 404 returned error can't find the container with id 453b86544165dd080a05a8bedeeb7fc86dad59a3af705f6912ce1915c6922910
Sep 18 21:06:04.384551 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.384496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:04.384647 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:04.384611    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:04.384689 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:04.384664    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret podName:5b2108fb-5c03-49d2-a864-2ef38a1fbaab nodeName:}" failed. No retries permitted until 2026-09-18 21:06:05.384646235 +0000 UTC m=+2.991705797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret") pod "global-pull-secret-syncer-rzd48" (UID: "5b2108fb-5c03-49d2-a864-2ef38a1fbaab") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:04.484926 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.484848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:04.485084 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:04.485016    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:04.485148 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:04.485087    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs podName:0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa nodeName:}" failed. No retries permitted until 2026-09-18 21:06:05.48506691 +0000 UTC m=+3.092126464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs") pod "network-metrics-daemon-bt5w7" (UID: "0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:04.585707 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.585659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmngj\" (UniqueName: \"kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj\") pod \"network-check-target-pm2mb\" (UID: \"085474f0-4cdb-4b45-9fd2-f00a4e10af1c\") " pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:04.585880 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:04.585838    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:06:04.585880 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:04.585866    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:06:04.585880 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:04.585881    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mmngj for pod openshift-network-diagnostics/network-check-target-pm2mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:04.586065 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:04.585940    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj podName:085474f0-4cdb-4b45-9fd2-f00a4e10af1c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:05.585921489 +0000 UTC m=+3.192981033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmngj" (UniqueName: "kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj") pod "network-check-target-pm2mb" (UID: "085474f0-4cdb-4b45-9fd2-f00a4e10af1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:04.755386 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.755200    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:06:04.918415 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.918314    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:04.918586 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:04.918520    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:04.955306 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.955222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-khjbr" event={"ID":"b57c9901-f383-4e37-b782-dfcfd75ccbc4","Type":"ContainerStarted","Data":"124d652b8b2c163d7ae46dc42245d11754c1c549e2cc88db13ff17129279ef88"}
Sep 18 21:06:04.958643 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.958605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-45p77" event={"ID":"fada2848-646b-456e-bd95-606b2da5c298","Type":"ContainerStarted","Data":"ed78811bf15e367b5841f4b77dcb2c5593882cb6cd96935785899005ccb7f2de"}
Sep 18 21:06:04.979572 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.979453    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 21:01:04 +0000 UTC" deadline="2028-06-11 00:59:11.365336588 +0000 UTC"
Sep 18 21:06:04.979572 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.979486    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15147h53m6.385855542s"
Sep 18 21:06:04.982336 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.982309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-27drt" event={"ID":"63bb4238-7a5e-462c-9b50-45e3b2813957","Type":"ContainerStarted","Data":"f6aa9c6e5e0e7f84d8b9a08d3ebb8a801af9ae796917af7b1042c421aaaa34c6"}
Sep 18 21:06:04.990673 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.989931    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rcfh2" event={"ID":"9187f979-a13d-4dc5-9fa2-362a99aaaee4","Type":"ContainerStarted","Data":"fcd95dd59419c7df7c42dd034d58c6d7e6e0fa1e08f67b132cc3dd56bebe59b0"}
Sep 18 21:06:04.999935 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:04.999060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr" event={"ID":"74da1b67-8d01-4ef9-9de1-7349d34cc6de","Type":"ContainerStarted","Data":"e3c9193e4403c1bce11fc3fec3db5de0b0252d59d44b25d810a1b03ef482eaa6"}
Sep 18 21:06:05.018386 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:05.018327    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal" event={"ID":"0cbd45bdbcda509ce01b35a6f5d64d3d","Type":"ContainerStarted","Data":"680704cdd1bfd4c88139e17183bf5ee0726a1697fb41faf2e77bd9480fae8697"}
Sep 18 21:06:05.033203 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:05.033105    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5" event={"ID":"9dda4277-1b86-4bad-8334-d9a06d7d9b3b","Type":"ContainerStarted","Data":"453b86544165dd080a05a8bedeeb7fc86dad59a3af705f6912ce1915c6922910"}
Sep 18 21:06:05.047034 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:05.046994    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tkw2z" event={"ID":"dc1f208a-bfe4-49d1-9dac-1214ace566d8","Type":"ContainerStarted","Data":"3de2d0075ffe3d1fc1595e901af20e8d923c8c8ec7d34900883c403aebeed6f1"}
Sep 18 21:06:05.055161 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:05.055137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gcgn5" event={"ID":"abfd23c2-a5d2-442e-a53b-ba3556428037","Type":"ContainerStarted","Data":"843b1e168b597f7f632fb054f8d08c8649daf94336c4f64dcddce56e11de5f76"}
Sep 18 21:06:05.067270 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:05.067234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-khlzv" event={"ID":"13cbaf64-d066-42f3-a262-8e56d161434a","Type":"ContainerStarted","Data":"e7cf24ae0dcd786253fde31fd9dd20a5267f707325c3d0ee570c838ebeeff996"}
Sep 18 21:06:05.095116 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:05.095062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal" event={"ID":"c6ac307dfcd56096fce875fa1e1ed1ca","Type":"ContainerStarted","Data":"0d2cb6b2810586b0d39017ac1ac28cae66b8a6e1815dc681cc6eae3f15c3ee82"}
Sep 18 21:06:05.393653 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:05.393554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:05.393834 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:05.393746    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:05.393834 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:05.393814    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret podName:5b2108fb-5c03-49d2-a864-2ef38a1fbaab nodeName:}" failed. No retries permitted until 2026-09-18 21:06:07.393794789 +0000 UTC m=+5.000854345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret") pod "global-pull-secret-syncer-rzd48" (UID: "5b2108fb-5c03-49d2-a864-2ef38a1fbaab") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:05.494825 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:05.494036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:05.494825 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:05.494198    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:05.494825 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:05.494276    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs podName:0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa nodeName:}" failed. No retries permitted until 2026-09-18 21:06:07.4942541 +0000 UTC m=+5.101313645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs") pod "network-metrics-daemon-bt5w7" (UID: "0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:05.595391 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:05.595349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmngj\" (UniqueName: \"kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj\") pod \"network-check-target-pm2mb\" (UID: \"085474f0-4cdb-4b45-9fd2-f00a4e10af1c\") " pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:05.595579 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:05.595523    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:06:05.595579 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:05.595547    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:06:05.595579 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:05.595560    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mmngj for pod openshift-network-diagnostics/network-check-target-pm2mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:05.595796 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:05.595628    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj podName:085474f0-4cdb-4b45-9fd2-f00a4e10af1c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:07.595601588 +0000 UTC m=+5.202661133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmngj" (UniqueName: "kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj") pod "network-check-target-pm2mb" (UID: "085474f0-4cdb-4b45-9fd2-f00a4e10af1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:05.916188 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:05.916152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:05.916700 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:05.916307    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:05.917837 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:05.917243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:05.917837 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:05.917371    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:05.980510 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:05.980447    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 21:01:04 +0000 UTC" deadline="2028-03-13 08:03:06.440904469 +0000 UTC"
Sep 18 21:06:05.980510 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:05.980510    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12994h57m0.460398743s"
Sep 18 21:06:06.108707 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:06.108452    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:06:06.572129 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:06.572093    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:06:06.918021 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:06.917945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:06.918461 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:06.918059    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:07.415829 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:07.415733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:07.416005 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:07.415940    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:07.416074 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:07.416010    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret podName:5b2108fb-5c03-49d2-a864-2ef38a1fbaab nodeName:}" failed. No retries permitted until 2026-09-18 21:06:11.415990797 +0000 UTC m=+9.023050362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret") pod "global-pull-secret-syncer-rzd48" (UID: "5b2108fb-5c03-49d2-a864-2ef38a1fbaab") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:07.516638 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:07.516602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:07.516819 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:07.516777    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:07.516900 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:07.516838    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs podName:0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa nodeName:}" failed. No retries permitted until 2026-09-18 21:06:11.516820639 +0000 UTC m=+9.123880191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs") pod "network-metrics-daemon-bt5w7" (UID: "0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:07.617666 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:07.617043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmngj\" (UniqueName: \"kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj\") pod \"network-check-target-pm2mb\" (UID: \"085474f0-4cdb-4b45-9fd2-f00a4e10af1c\") " pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:07.617666 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:07.617243    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:06:07.617666 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:07.617266    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:06:07.617666 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:07.617278    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mmngj for pod openshift-network-diagnostics/network-check-target-pm2mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:07.617666 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:07.617332    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj podName:085474f0-4cdb-4b45-9fd2-f00a4e10af1c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:11.617315087 +0000 UTC m=+9.224374649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmngj" (UniqueName: "kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj") pod "network-check-target-pm2mb" (UID: "085474f0-4cdb-4b45-9fd2-f00a4e10af1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:07.916622 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:07.915857    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:07.916622 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:07.915908    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:07.916622 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:07.915992    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:07.916622 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:07.916520    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:08.916689 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:08.916175    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:08.916689 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:08.916316    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:09.915557 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:09.915524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:09.915749 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:09.915663    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:09.916211 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:09.916026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:09.916211 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:09.916125    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:10.916421 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:10.916390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:10.916876 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:10.916528    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:11.447560 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:11.447516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:11.447732 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:11.447682    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:11.447799 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:11.447738    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret podName:5b2108fb-5c03-49d2-a864-2ef38a1fbaab nodeName:}" failed. No retries permitted until 2026-09-18 21:06:19.447721833 +0000 UTC m=+17.054781376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret") pod "global-pull-secret-syncer-rzd48" (UID: "5b2108fb-5c03-49d2-a864-2ef38a1fbaab") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:11.548497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:11.548464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:11.548670 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:11.548640    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:11.548737 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:11.548721    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs podName:0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa nodeName:}" failed. No retries permitted until 2026-09-18 21:06:19.548701268 +0000 UTC m=+17.155760810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs") pod "network-metrics-daemon-bt5w7" (UID: "0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:11.649099 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:11.649022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmngj\" (UniqueName: \"kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj\") pod \"network-check-target-pm2mb\" (UID: \"085474f0-4cdb-4b45-9fd2-f00a4e10af1c\") " pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:11.649238 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:11.649182    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:06:11.649238 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:11.649209    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:06:11.649238 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:11.649223    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mmngj for pod openshift-network-diagnostics/network-check-target-pm2mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:11.649397 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:11.649277    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj podName:085474f0-4cdb-4b45-9fd2-f00a4e10af1c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:19.649260394 +0000 UTC m=+17.256319945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmngj" (UniqueName: "kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj") pod "network-check-target-pm2mb" (UID: "085474f0-4cdb-4b45-9fd2-f00a4e10af1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:11.915859 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:11.915732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:11.915859 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:11.915762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:11.916082 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:11.915874    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:11.916210 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:11.916170    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:12.917280 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:12.916806    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:12.917280 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:12.916937    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:13.915941 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:13.915866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:13.915941 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:13.915911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:13.916115 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:13.915997    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:13.916169 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:13.916130    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:14.915821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:14.915788    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:14.916256 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:14.915929    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:15.915906 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:15.915833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:15.916423 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:15.915936    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:15.916423 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:15.915971    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:15.916423 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:15.916035    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:16.915475 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:16.915439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:16.915682 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:16.915566    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:17.916102 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:17.916073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:17.916507 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:17.916157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:17.916507 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:17.916196    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:17.916507 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:17.916228    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:18.916339 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:18.916303    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:18.916637 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:18.916450    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:19.509247 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:19.509176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:19.509414 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:19.509312    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:19.509414 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:19.509380    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret podName:5b2108fb-5c03-49d2-a864-2ef38a1fbaab nodeName:}" failed. No retries permitted until 2026-09-18 21:06:35.509363015 +0000 UTC m=+33.116422555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret") pod "global-pull-secret-syncer-rzd48" (UID: "5b2108fb-5c03-49d2-a864-2ef38a1fbaab") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:19.610471 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:19.610441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:19.610634 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:19.610579    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:19.610634 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:19.610629    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs podName:0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa nodeName:}" failed. No retries permitted until 2026-09-18 21:06:35.610615783 +0000 UTC m=+33.217675324 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs") pod "network-metrics-daemon-bt5w7" (UID: "0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:19.710823 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:19.710782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmngj\" (UniqueName: \"kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj\") pod \"network-check-target-pm2mb\" (UID: \"085474f0-4cdb-4b45-9fd2-f00a4e10af1c\") " pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:19.710989 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:19.710927    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:06:19.710989 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:19.710942    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:06:19.710989 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:19.710952    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mmngj for pod openshift-network-diagnostics/network-check-target-pm2mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:19.711138 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:19.711011    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj podName:085474f0-4cdb-4b45-9fd2-f00a4e10af1c nodeName:}" failed. No retries permitted until 2026-09-18 21:06:35.710994393 +0000 UTC m=+33.318053954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmngj" (UniqueName: "kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj") pod "network-check-target-pm2mb" (UID: "085474f0-4cdb-4b45-9fd2-f00a4e10af1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:19.916356 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:19.916284    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:19.916779 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:19.916289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:19.916779 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:19.916387    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:19.916779 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:19.916473    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:20.915778 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:20.915700    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:20.915957 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:20.915844    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:21.915798 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:21.915762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:21.916235 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:21.915877    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:21.916235 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:21.915935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:21.916235 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:21.916072    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:22.916600 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:22.916570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:22.916969 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:22.916672    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:23.147290 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:23.147147    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal" event={"ID":"c6ac307dfcd56096fce875fa1e1ed1ca","Type":"ContainerStarted","Data":"3e774a3141783e63693060f452950b401aff38b8ddfe24b934481de4cb13c441"}
Sep 18 21:06:23.147382 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:23.147310    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:23.160179 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:23.160164    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 21:06:23.162517 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:23.162498    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal"]
Sep 18 21:06:23.916213 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:23.916033    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:23.916329 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:23.916082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:23.916329 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:23.916291    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:23.916427 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:23.916369    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:24.150847 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.150816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-khlzv" event={"ID":"13cbaf64-d066-42f3-a262-8e56d161434a","Type":"ContainerStarted","Data":"c81b9fba67f8354dfb7cf89e198736bb2b4684bf29e78a4f1d3715df2fa4b8f0"}
Sep 18 21:06:24.152330 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.152308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-45p77" event={"ID":"fada2848-646b-456e-bd95-606b2da5c298","Type":"ContainerStarted","Data":"4786fbe832012c4ae4952ec0a6daf7e1c625faecb5c64e86e2a223c755ef9875"}
Sep 18 21:06:24.153762 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.153737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-27drt" event={"ID":"63bb4238-7a5e-462c-9b50-45e3b2813957","Type":"ContainerStarted","Data":"d58faad06b99240266a7974ea06ca29a6a91d0d1d5a56f64ce750a8d6f0d4649"}
Sep 18 21:06:24.155151 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.155124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rcfh2" event={"ID":"9187f979-a13d-4dc5-9fa2-362a99aaaee4","Type":"ContainerStarted","Data":"c16aed66eef8354ed99284c648cfa99f20a65138f729a4ced05e5be24564b096"}
Sep 18 21:06:24.156691 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.156670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr" event={"ID":"74da1b67-8d01-4ef9-9de1-7349d34cc6de","Type":"ContainerStarted","Data":"379ce924b2baeb3f85cc3925f785c20d379c8e671f60162116442d7730569423"}
Sep 18 21:06:24.157986 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.157962    2579 generic.go:356] "Generic (PLEG): container finished" podID="0cbd45bdbcda509ce01b35a6f5d64d3d" containerID="87ec26860c8e03cd721f5fd0f7e4341cee748e1dd128b0e3d8d0fb0f0d0a288a" exitCode=0
Sep 18 21:06:24.158062 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.158000    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal" event={"ID":"0cbd45bdbcda509ce01b35a6f5d64d3d","Type":"ContainerDied","Data":"87ec26860c8e03cd721f5fd0f7e4341cee748e1dd128b0e3d8d0fb0f0d0a288a"}
Sep 18 21:06:24.160840 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.160825    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:06:24.161170 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.161153    2579 generic.go:356] "Generic (PLEG): container finished" podID="9dda4277-1b86-4bad-8334-d9a06d7d9b3b" containerID="87f8ec6727804deca4199e115b3cebbe707798b562daf9f757cd6ef606a4b80b" exitCode=1
Sep 18 21:06:24.161226 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.161216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5" event={"ID":"9dda4277-1b86-4bad-8334-d9a06d7d9b3b","Type":"ContainerStarted","Data":"6a56f80e972702feb7826b1872f194496501c4d15a0a8acb9d38465f7c447f79"}
Sep 18 21:06:24.161273 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.161263    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5" event={"ID":"9dda4277-1b86-4bad-8334-d9a06d7d9b3b","Type":"ContainerStarted","Data":"fb890d35bcf3cf74479a6db1d29963656e570e4e83b2ca1dc66b34d5904ba2e4"}
Sep 18 21:06:24.161321 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.161306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5" event={"ID":"9dda4277-1b86-4bad-8334-d9a06d7d9b3b","Type":"ContainerStarted","Data":"bc622edf42ad72298463ee58a63748999c36bfe15992adb4ed394b1aa5d6ac85"}
Sep 18 21:06:24.161356 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.161342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5" event={"ID":"9dda4277-1b86-4bad-8334-d9a06d7d9b3b","Type":"ContainerStarted","Data":"79c3e0c36808d93e12284b2da8f79b5b129826d2556acb1d60bb9308e464d19c"}
Sep 18 21:06:24.161401 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.161374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5" event={"ID":"9dda4277-1b86-4bad-8334-d9a06d7d9b3b","Type":"ContainerDied","Data":"87f8ec6727804deca4199e115b3cebbe707798b562daf9f757cd6ef606a4b80b"}
Sep 18 21:06:24.161439 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.161413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5" event={"ID":"9dda4277-1b86-4bad-8334-d9a06d7d9b3b","Type":"ContainerStarted","Data":"e54f73451ca571496b0119873292fa0de507b0b5608ea3eb90cbf54430f6a9e7"}
Sep 18 21:06:24.162525 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.162507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tkw2z" event={"ID":"dc1f208a-bfe4-49d1-9dac-1214ace566d8","Type":"ContainerStarted","Data":"22af5bce676d9f75394d6b44e16a6d633353aa3fc45e208b655176134edaedec"}
Sep 18 21:06:24.163923 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.163904    2579 generic.go:356] "Generic (PLEG): container finished" podID="abfd23c2-a5d2-442e-a53b-ba3556428037" containerID="b7d5a1e930beeb873aff82bb6fbbe8c333402e1fb3e7c34947f018e62ab64c6f" exitCode=0
Sep 18 21:06:24.163994 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.163975    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gcgn5" event={"ID":"abfd23c2-a5d2-442e-a53b-ba3556428037","Type":"ContainerDied","Data":"b7d5a1e930beeb873aff82bb6fbbe8c333402e1fb3e7c34947f018e62ab64c6f"}
Sep 18 21:06:24.164185 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.164174    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:24.178773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.178737    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-khlzv" podStartSLOduration=2.372252112 podStartE2EDuration="21.178721898s" totalImagesPullingTime="18.806469786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:24.178630363 +0000 UTC m=+21.785689921" watchObservedRunningTime="2026-09-18 21:06:24.178721898 +0000 UTC m=+21.785781456"
Sep 18 21:06:24.191764 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.191746    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 21:06:24.191847 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:24.191790    2579 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-92.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal"
Sep 18 21:06:24.206993 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.206942    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-92.ec2.internal" podStartSLOduration=1.2069247 podStartE2EDuration="1.2069247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:24.205864187 +0000 UTC m=+21.812923747" watchObservedRunningTime="2026-09-18 21:06:24.2069247 +0000 UTC m=+21.813984259"
Sep 18 21:06:24.326048 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.325890    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tkw2z" podStartSLOduration=2.604505516 podStartE2EDuration="21.325872658s" totalImagesPullingTime="18.721367142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:24.292691892 +0000 UTC m=+21.899751449" watchObservedRunningTime="2026-09-18 21:06:24.325872658 +0000 UTC m=+21.932932219"
Sep 18 21:06:24.394582 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.394539    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-27drt" podStartSLOduration=2.6559411859999997 podStartE2EDuration="21.394523209s" totalImagesPullingTime="18.738582023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:24.36043671 +0000 UTC m=+21.967496267" watchObservedRunningTime="2026-09-18 21:06:24.394523209 +0000 UTC m=+22.001582768"
Sep 18 21:06:24.394708 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.394692    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-45p77" podStartSLOduration=2.003793095 podStartE2EDuration="21.394687037s" totalImagesPullingTime="19.390893942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:24.394498103 +0000 UTC m=+22.001557661" watchObservedRunningTime="2026-09-18 21:06:24.394687037 +0000 UTC m=+22.001746591"
Sep 18 21:06:24.468924 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.468889    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rcfh2" podStartSLOduration=2.647845994 podStartE2EDuration="21.468876717s" totalImagesPullingTime="18.821030723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:24.468567269 +0000 UTC m=+22.075626828" watchObservedRunningTime="2026-09-18 21:06:24.468876717 +0000 UTC m=+22.075936272"
Sep 18 21:06:24.916549 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:24.916509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:24.916713 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:24.916665    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:25.011829 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:25.011738    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 21:06:25.168344 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:25.168272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr" event={"ID":"74da1b67-8d01-4ef9-9de1-7349d34cc6de","Type":"ContainerStarted","Data":"2bc4bb9a69bddd1881fdd780e8a919df0c394d9c9a3da62164e7e5d994e4900d"}
Sep 18 21:06:25.170301 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:25.170276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal" event={"ID":"0cbd45bdbcda509ce01b35a6f5d64d3d","Type":"ContainerStarted","Data":"d9a96815f2ab8daccfb5f1d07204adeb0d59e0c81775aa1f784d4ca8889e6167"}
Sep 18 21:06:25.172040 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:25.172003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-khjbr" event={"ID":"b57c9901-f383-4e37-b782-dfcfd75ccbc4","Type":"ContainerStarted","Data":"1814a769d519defb13b3cd89f9dfb04c086f15ce88974692caa0d3592efad2bb"}
Sep 18 21:06:25.200925 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:25.200860    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-92.ec2.internal" podStartSLOduration=22.200837378 podStartE2EDuration="22.200837378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:25.200604897 +0000 UTC m=+22.807664455" watchObservedRunningTime="2026-09-18 21:06:25.200837378 +0000 UTC m=+22.807896936"
Sep 18 21:06:25.444608 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:25.444519    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-27drt"
Sep 18 21:06:25.445344 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:25.445322    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-27drt"
Sep 18 21:06:25.499121 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:25.499063    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-khjbr" podStartSLOduration=3.784371265 podStartE2EDuration="22.499043723s" totalImagesPullingTime="18.714672458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:25.244092521 +0000 UTC m=+22.851152080" watchObservedRunningTime="2026-09-18 21:06:25.499043723 +0000 UTC m=+23.106103282"
Sep 18 21:06:25.836782 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:25.836610    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T21:06:25.01175469Z","UUID":"f8be7d6d-cacf-451d-a2e9-6f4ed530ab9e","Handler":null,"Name":"","Endpoint":""}
Sep 18 21:06:25.839490 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:25.839463    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 21:06:25.839659 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:25.839504    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 21:06:25.915766 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:25.915734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:25.915954 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:25.915734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:25.915954 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:25.915855    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:25.916065 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:25.916007    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:26.916112 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:26.916087    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:26.916625 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:26.916193    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:27.178827 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:27.178729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr" event={"ID":"74da1b67-8d01-4ef9-9de1-7349d34cc6de","Type":"ContainerStarted","Data":"153a89da5e51bcb7e7bddf65e9aa766b172aa638531a5e9f7ec4269dd0abb680"}
Sep 18 21:06:27.182601 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:27.182577    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:06:27.183062 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:27.183045    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 21:06:27.183062 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:27.183051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5" event={"ID":"9dda4277-1b86-4bad-8334-d9a06d7d9b3b","Type":"ContainerStarted","Data":"320b9f554598072d346b9ae1ee78440ac16a38184fff84632f2ae33d567918b7"}
Sep 18 21:06:27.915810 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:27.915780    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:27.915975 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:27.915900    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:27.915975 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:27.915947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:27.916067 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:27.916034    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:28.915988 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:28.915807    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:28.916418 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:28.916080    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:29.190278 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:29.190205    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:06:29.190681 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:29.190631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5" event={"ID":"9dda4277-1b86-4bad-8334-d9a06d7d9b3b","Type":"ContainerStarted","Data":"6953b22a206d1245fb16cd666c9c4015d3d61fd3300cd65be69a67d97b707569"}
Sep 18 21:06:29.190928 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:29.190901    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:29.191036 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:29.190970    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:29.191094 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:29.191085    2579 scope.go:117] "RemoveContainer" containerID="87f8ec6727804deca4199e115b3cebbe707798b562daf9f757cd6ef606a4b80b"
Sep 18 21:06:29.192692 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:29.192668    2579 generic.go:356] "Generic (PLEG): container finished" podID="abfd23c2-a5d2-442e-a53b-ba3556428037" containerID="95db263caa6512a7fc765617136cf70817c191fe36968095f839dd8ed0cdb171" exitCode=0
Sep 18 21:06:29.192801 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:29.192706    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gcgn5" event={"ID":"abfd23c2-a5d2-442e-a53b-ba3556428037","Type":"ContainerDied","Data":"95db263caa6512a7fc765617136cf70817c191fe36968095f839dd8ed0cdb171"}
Sep 18 21:06:29.206933 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:29.206915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:29.239146 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:29.239104    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc7mr" podStartSLOduration=9.398957433 podStartE2EDuration="26.239087216s" totalImagesPullingTime="16.840129783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:27.236818866 +0000 UTC m=+24.843878425" watchObservedRunningTime="2026-09-18 21:06:29.239087216 +0000 UTC m=+26.846146781"
Sep 18 21:06:29.916079 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:29.916050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:29.916609 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:29.916086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:29.916609 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:29.916160    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:29.916609 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:29.916267    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:30.079435 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.079400    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-27drt"
Sep 18 21:06:30.079553 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.079514    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 21:06:30.080014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.079998    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-27drt"
Sep 18 21:06:30.199082 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.199065    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:06:30.199428 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.199405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5" event={"ID":"9dda4277-1b86-4bad-8334-d9a06d7d9b3b","Type":"ContainerStarted","Data":"d685356bf0be3e54bd60d8794d3ece5f17c10e0e3ed42e0c9375786f7f800788"}
Sep 18 21:06:30.199695 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.199674    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:30.214094 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.214073    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:06:30.262663 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.262613    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5" podStartSLOduration=8.487597605 podStartE2EDuration="27.262596707s" totalImagesPullingTime="18.774999102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:30.254281389 +0000 UTC m=+27.861340946" watchObservedRunningTime="2026-09-18 21:06:30.262596707 +0000 UTC m=+27.869656243"
Sep 18 21:06:30.303503 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.303474    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bt5w7"]
Sep 18 21:06:30.303623 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.303610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:30.303771 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:30.303743    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:30.304209 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.304187    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pm2mb"]
Sep 18 21:06:30.304281 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.304274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:30.304364 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:30.304344    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:30.311262 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.311242    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rzd48"]
Sep 18 21:06:30.311348 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:30.311326    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:30.311411 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:30.311395    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:31.203916 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:31.203881    2579 generic.go:356] "Generic (PLEG): container finished" podID="abfd23c2-a5d2-442e-a53b-ba3556428037" containerID="5bcc4c5e1bd743c3e70d672194fc7007a2955922d32a2679784dd0423361d7a7" exitCode=0
Sep 18 21:06:31.204343 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:31.203960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gcgn5" event={"ID":"abfd23c2-a5d2-442e-a53b-ba3556428037","Type":"ContainerDied","Data":"5bcc4c5e1bd743c3e70d672194fc7007a2955922d32a2679784dd0423361d7a7"}
Sep 18 21:06:31.915993 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:31.915964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:31.916166 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:31.916093    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:31.916166 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:31.916139    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:31.916166 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:31.916154    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:31.916327 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:31.916272    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:31.916381 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:31.916367    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:32.208679 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:32.208454    2579 generic.go:356] "Generic (PLEG): container finished" podID="abfd23c2-a5d2-442e-a53b-ba3556428037" containerID="b98476bc32adf9f65a343cf3d5357490608ea98defc557af9b591b5555b5d2c1" exitCode=0
Sep 18 21:06:32.208973 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:32.208533    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gcgn5" event={"ID":"abfd23c2-a5d2-442e-a53b-ba3556428037","Type":"ContainerDied","Data":"b98476bc32adf9f65a343cf3d5357490608ea98defc557af9b591b5555b5d2c1"}
Sep 18 21:06:33.915995 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:33.915962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:33.916425 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:33.916013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:33.916425 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:33.915972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:33.916425 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:33.916090    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:33.916425 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:33.916304    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:33.916425 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:33.916392    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:35.531437 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:35.531364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:35.531937 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:35.531512    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:35.531937 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:35.531585    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret podName:5b2108fb-5c03-49d2-a864-2ef38a1fbaab nodeName:}" failed. No retries permitted until 2026-09-18 21:07:07.531567605 +0000 UTC m=+65.138627161 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret") pod "global-pull-secret-syncer-rzd48" (UID: "5b2108fb-5c03-49d2-a864-2ef38a1fbaab") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:06:35.632629 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:35.632594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:35.632784 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:35.632748    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:35.632841 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:35.632818    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs podName:0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa nodeName:}" failed. No retries permitted until 2026-09-18 21:07:07.632797718 +0000 UTC m=+65.239857274 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs") pod "network-metrics-daemon-bt5w7" (UID: "0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:06:35.733947 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:35.733910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmngj\" (UniqueName: \"kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj\") pod \"network-check-target-pm2mb\" (UID: \"085474f0-4cdb-4b45-9fd2-f00a4e10af1c\") " pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:35.734108 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:35.734050    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:06:35.734108 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:35.734074    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:06:35.734108 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:35.734089    2579 projected.go:196] Error preparing data for projected volume kube-api-access-mmngj for pod openshift-network-diagnostics/network-check-target-pm2mb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:35.734246 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:35.734144    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj podName:085474f0-4cdb-4b45-9fd2-f00a4e10af1c nodeName:}" failed. No retries permitted until 2026-09-18 21:07:07.734131854 +0000 UTC m=+65.341191390 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmngj" (UniqueName: "kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj") pod "network-check-target-pm2mb" (UID: "085474f0-4cdb-4b45-9fd2-f00a4e10af1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:06:35.915977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:35.915898    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:35.916124 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:35.915898    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:35.916124 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:35.916019    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzd48" podUID="5b2108fb-5c03-49d2-a864-2ef38a1fbaab"
Sep 18 21:06:35.916124 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:35.916035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:35.916253 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:35.916150    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm2mb" podUID="085474f0-4cdb-4b45-9fd2-f00a4e10af1c"
Sep 18 21:06:35.916317 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:35.916287    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bt5w7" podUID="0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa"
Sep 18 21:06:36.605310 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.605259    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-92.ec2.internal" event="NodeReady"
Sep 18 21:06:36.605676 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.605382    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 21:06:36.655461 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.655438    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84dd74ddb9-nbd9b"]
Sep 18 21:06:36.672550 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.672531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.675618 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.675592    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 21:06:36.675715 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.675626    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 21:06:36.676229 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.676217    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 21:06:36.676331 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.676317    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d7nd7\""
Sep 18 21:06:36.679881 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.679862    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84dd74ddb9-nbd9b"]
Sep 18 21:06:36.684094 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.684076    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 21:06:36.687988 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.687968    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4brns"]
Sep 18 21:06:36.710541 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.710519    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r9rc6"]
Sep 18 21:06:36.710679 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.710611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4brns"
Sep 18 21:06:36.713453 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.713436    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 21:06:36.713453 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.713442    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 21:06:36.713696 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.713683    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 21:06:36.713861 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.713846    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jdg2h\""
Sep 18 21:06:36.734544 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.734526    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9rc6"]
Sep 18 21:06:36.734618 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.734561    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4brns"]
Sep 18 21:06:36.734689 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.734628    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:36.738135 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.738119    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 21:06:36.738493 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.738478    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4xcf\""
Sep 18 21:06:36.738574 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.738494    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 21:06:36.842048 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/203754e1-73e8-415d-be7a-f187ea164375-cert\") pod \"ingress-canary-4brns\" (UID: \"203754e1-73e8-415d-be7a-f187ea164375\") " pod="openshift-ingress-canary/ingress-canary-4brns"
Sep 18 21:06:36.842048 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1225d31-f513-4cb7-b21a-a97accdb1bfb-tmp-dir\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:36.842255 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-bound-sa-token\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.842255 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-certificates\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.842255 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.842255 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1225d31-f513-4cb7-b21a-a97accdb1bfb-config-volume\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:36.842255 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ngmt\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-kube-api-access-8ngmt\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.842255 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:36.842255 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dw6\" (UniqueName: \"kubernetes.io/projected/d1225d31-f513-4cb7-b21a-a97accdb1bfb-kube-api-access-h2dw6\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:36.842496 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/abcc2145-3a92-486e-aaa8-16b8551f6bfe-trusted-ca\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.842496 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzbl\" (UniqueName: \"kubernetes.io/projected/203754e1-73e8-415d-be7a-f187ea164375-kube-api-access-wkzbl\") pod \"ingress-canary-4brns\" (UID: \"203754e1-73e8-415d-be7a-f187ea164375\") " pod="openshift-ingress-canary/ingress-canary-4brns"
Sep 18 21:06:36.842496 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/abcc2145-3a92-486e-aaa8-16b8551f6bfe-ca-trust-extracted\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.842496 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/abcc2145-3a92-486e-aaa8-16b8551f6bfe-image-registry-private-configuration\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.842496 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.842340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/abcc2145-3a92-486e-aaa8-16b8551f6bfe-installation-pull-secrets\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.943316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/abcc2145-3a92-486e-aaa8-16b8551f6bfe-image-registry-private-configuration\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.943316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/abcc2145-3a92-486e-aaa8-16b8551f6bfe-installation-pull-secrets\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.943316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/203754e1-73e8-415d-be7a-f187ea164375-cert\") pod \"ingress-canary-4brns\" (UID: \"203754e1-73e8-415d-be7a-f187ea164375\") " pod="openshift-ingress-canary/ingress-canary-4brns"
Sep 18 21:06:36.943528 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1225d31-f513-4cb7-b21a-a97accdb1bfb-tmp-dir\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:36.943528 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-bound-sa-token\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.943528 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-certificates\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.943528 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.943528 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1225d31-f513-4cb7-b21a-a97accdb1bfb-config-volume\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:36.943528 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ngmt\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-kube-api-access-8ngmt\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.943528 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:36.943930 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dw6\" (UniqueName: \"kubernetes.io/projected/d1225d31-f513-4cb7-b21a-a97accdb1bfb-kube-api-access-h2dw6\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:36.943930 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/abcc2145-3a92-486e-aaa8-16b8551f6bfe-trusted-ca\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.943930 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzbl\" (UniqueName: \"kubernetes.io/projected/203754e1-73e8-415d-be7a-f187ea164375-kube-api-access-wkzbl\") pod \"ingress-canary-4brns\" (UID: \"203754e1-73e8-415d-be7a-f187ea164375\") " pod="openshift-ingress-canary/ingress-canary-4brns"
Sep 18 21:06:36.943930 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.943624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/abcc2145-3a92-486e-aaa8-16b8551f6bfe-ca-trust-extracted\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.943930 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:36.943775    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:06:36.943930 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:36.943861    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls podName:d1225d31-f513-4cb7-b21a-a97accdb1bfb nodeName:}" failed. No retries permitted until 2026-09-18 21:06:37.443837425 +0000 UTC m=+35.050896975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls") pod "dns-default-r9rc6" (UID: "d1225d31-f513-4cb7-b21a-a97accdb1bfb") : secret "dns-default-metrics-tls" not found
Sep 18 21:06:36.944218 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.944059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-certificates\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.944218 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:36.944170    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:06:36.944218 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:36.944187    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84dd74ddb9-nbd9b: secret "image-registry-tls" not found
Sep 18 21:06:36.944368 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:36.944245    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls podName:abcc2145-3a92-486e-aaa8-16b8551f6bfe nodeName:}" failed. No retries permitted until 2026-09-18 21:06:37.444228689 +0000 UTC m=+35.051288231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls") pod "image-registry-84dd74ddb9-nbd9b" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe") : secret "image-registry-tls" not found
Sep 18 21:06:36.944728 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.944572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/abcc2145-3a92-486e-aaa8-16b8551f6bfe-ca-trust-extracted\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.944728 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.944684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/abcc2145-3a92-486e-aaa8-16b8551f6bfe-trusted-ca\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.947618 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.947519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/abcc2145-3a92-486e-aaa8-16b8551f6bfe-image-registry-private-configuration\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.947747 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.947524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/203754e1-73e8-415d-be7a-f187ea164375-cert\") pod \"ingress-canary-4brns\" (UID: \"203754e1-73e8-415d-be7a-f187ea164375\") " pod="openshift-ingress-canary/ingress-canary-4brns"
Sep 18 21:06:36.947747 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.947573    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/abcc2145-3a92-486e-aaa8-16b8551f6bfe-installation-pull-secrets\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.956334 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.956307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1225d31-f513-4cb7-b21a-a97accdb1bfb-tmp-dir\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:36.956839 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.956815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1225d31-f513-4cb7-b21a-a97accdb1bfb-config-volume\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:36.957933 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.957902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ngmt\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-kube-api-access-8ngmt\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.958026 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.957990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-bound-sa-token\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:36.958240 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.958222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzbl\" (UniqueName: \"kubernetes.io/projected/203754e1-73e8-415d-be7a-f187ea164375-kube-api-access-wkzbl\") pod \"ingress-canary-4brns\" (UID: \"203754e1-73e8-415d-be7a-f187ea164375\") " pod="openshift-ingress-canary/ingress-canary-4brns"
Sep 18 21:06:36.958332 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:36.958315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dw6\" (UniqueName: \"kubernetes.io/projected/d1225d31-f513-4cb7-b21a-a97accdb1bfb-kube-api-access-h2dw6\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:37.019243 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.019225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4brns"
Sep 18 21:06:37.223326 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.223261    2579 generic.go:356] "Generic (PLEG): container finished" podID="abfd23c2-a5d2-442e-a53b-ba3556428037" containerID="25631058e548cf9f115a71608bbfa0415038eb286ad8e951f7dfd5f452e0dc47" exitCode=0
Sep 18 21:06:37.223441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.223328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gcgn5" event={"ID":"abfd23c2-a5d2-442e-a53b-ba3556428037","Type":"ContainerDied","Data":"25631058e548cf9f115a71608bbfa0415038eb286ad8e951f7dfd5f452e0dc47"}
Sep 18 21:06:37.335623 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.335591    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4brns"]
Sep 18 21:06:37.338178 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:06:37.338152    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203754e1_73e8_415d_be7a_f187ea164375.slice/crio-d61846efd754ebb23f307f4045cd644e6eaa873804571fa8916c0abe30cbdcd3 WatchSource:0}: Error finding container d61846efd754ebb23f307f4045cd644e6eaa873804571fa8916c0abe30cbdcd3: Status 404 returned error can't find the container with id d61846efd754ebb23f307f4045cd644e6eaa873804571fa8916c0abe30cbdcd3
Sep 18 21:06:37.446851 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.446827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:37.446951 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.446861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:37.446987 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:37.446949    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:06:37.446987 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:37.446956    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:06:37.446987 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:37.446973    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84dd74ddb9-nbd9b: secret "image-registry-tls" not found
Sep 18 21:06:37.447077 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:37.447007    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls podName:d1225d31-f513-4cb7-b21a-a97accdb1bfb nodeName:}" failed. No retries permitted until 2026-09-18 21:06:38.446993309 +0000 UTC m=+36.054052845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls") pod "dns-default-r9rc6" (UID: "d1225d31-f513-4cb7-b21a-a97accdb1bfb") : secret "dns-default-metrics-tls" not found
Sep 18 21:06:37.447077 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:37.447020    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls podName:abcc2145-3a92-486e-aaa8-16b8551f6bfe nodeName:}" failed. No retries permitted until 2026-09-18 21:06:38.44701421 +0000 UTC m=+36.054073745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls") pod "image-registry-84dd74ddb9-nbd9b" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe") : secret "image-registry-tls" not found
Sep 18 21:06:37.915613 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.915583    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:06:37.916353 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.915583    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:06:37.916353 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.915583    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:06:37.920513 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.920489    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 21:06:37.921363 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.921180    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 21:06:37.921363 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.921209    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 21:06:37.921363 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.921214    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xrfdl\""
Sep 18 21:06:37.921363 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.921230    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 21:06:37.921770 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:37.921753    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w6vrc\""
Sep 18 21:06:38.229744 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:38.229661    2579 generic.go:356] "Generic (PLEG): container finished" podID="abfd23c2-a5d2-442e-a53b-ba3556428037" containerID="204ef5538e55e99b806f8864a5c17f5120383d1bf48ab47824876588aed6ba61" exitCode=0
Sep 18 21:06:38.229891 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:38.229739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gcgn5" event={"ID":"abfd23c2-a5d2-442e-a53b-ba3556428037","Type":"ContainerDied","Data":"204ef5538e55e99b806f8864a5c17f5120383d1bf48ab47824876588aed6ba61"}
Sep 18 21:06:38.231074 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:38.231051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4brns" event={"ID":"203754e1-73e8-415d-be7a-f187ea164375","Type":"ContainerStarted","Data":"d61846efd754ebb23f307f4045cd644e6eaa873804571fa8916c0abe30cbdcd3"}
Sep 18 21:06:38.454561 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:38.454536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:38.454702 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:38.454638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:38.454765 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:38.454698    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:06:38.454765 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:38.454737    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:06:38.454765 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:38.454752    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84dd74ddb9-nbd9b: secret "image-registry-tls" not found
Sep 18 21:06:38.454882 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:38.454766    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls podName:d1225d31-f513-4cb7-b21a-a97accdb1bfb nodeName:}" failed. No retries permitted until 2026-09-18 21:06:40.454745571 +0000 UTC m=+38.061805128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls") pod "dns-default-r9rc6" (UID: "d1225d31-f513-4cb7-b21a-a97accdb1bfb") : secret "dns-default-metrics-tls" not found
Sep 18 21:06:38.454882 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:38.454787    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls podName:abcc2145-3a92-486e-aaa8-16b8551f6bfe nodeName:}" failed. No retries permitted until 2026-09-18 21:06:40.454776574 +0000 UTC m=+38.061836128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls") pod "image-registry-84dd74ddb9-nbd9b" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe") : secret "image-registry-tls" not found
Sep 18 21:06:39.240594 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:39.240558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gcgn5" event={"ID":"abfd23c2-a5d2-442e-a53b-ba3556428037","Type":"ContainerStarted","Data":"e2dd267d8407f96c567094c1f464f5f21b3a1dd3a4935ae0d01e7abf2f19ec80"}
Sep 18 21:06:39.269865 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:39.269785    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gcgn5" podStartSLOduration=7.686489126 podStartE2EDuration="36.269767569s" totalImagesPullingTime="28.583278443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:39.268314068 +0000 UTC m=+36.875373625" watchObservedRunningTime="2026-09-18 21:06:39.269767569 +0000 UTC m=+36.876827128"
Sep 18 21:06:39.418135 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:39.418101    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:39.515569 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:39.515373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:40.244499 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:40.244423    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4brns" event={"ID":"203754e1-73e8-415d-be7a-f187ea164375","Type":"ContainerStarted","Data":"f7c29ef5bd29cd0c2287f80c775ddcd1fab03309284b92f534547a4c524fcdbb"}
Sep 18 21:06:40.261781 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:40.261733    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4brns" podStartSLOduration=1.93397894 podStartE2EDuration="4.261712268s" totalImagesPullingTime="2.327733328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:06:40.26145472 +0000 UTC m=+37.868514279" watchObservedRunningTime="2026-09-18 21:06:40.261712268 +0000 UTC m=+37.868771825"
Sep 18 21:06:40.471740 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:40.471716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:40.471873 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:40.471750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:40.471873 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:40.471826    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:06:40.471873 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:40.471853    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:06:40.471873 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:40.471874    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls podName:d1225d31-f513-4cb7-b21a-a97accdb1bfb nodeName:}" failed. No retries permitted until 2026-09-18 21:06:44.471860258 +0000 UTC m=+42.078919795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls") pod "dns-default-r9rc6" (UID: "d1225d31-f513-4cb7-b21a-a97accdb1bfb") : secret "dns-default-metrics-tls" not found
Sep 18 21:06:40.472014 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:40.471876    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84dd74ddb9-nbd9b: secret "image-registry-tls" not found
Sep 18 21:06:40.472014 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:40.471920    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls podName:abcc2145-3a92-486e-aaa8-16b8551f6bfe nodeName:}" failed. No retries permitted until 2026-09-18 21:06:44.471906032 +0000 UTC m=+42.078965573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls") pod "image-registry-84dd74ddb9-nbd9b" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe") : secret "image-registry-tls" not found
Sep 18 21:06:41.643684 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:41.643638    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:41.814429 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:41.814399    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:44.500186 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:44.500151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:44.500186 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:44.500191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:44.500571 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:44.500295    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:06:44.500571 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:44.500296    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:06:44.500571 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:44.500317    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84dd74ddb9-nbd9b: secret "image-registry-tls" not found
Sep 18 21:06:44.500571 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:44.500348    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls podName:d1225d31-f513-4cb7-b21a-a97accdb1bfb nodeName:}" failed. No retries permitted until 2026-09-18 21:06:52.500333012 +0000 UTC m=+50.107392566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls") pod "dns-default-r9rc6" (UID: "d1225d31-f513-4cb7-b21a-a97accdb1bfb") : secret "dns-default-metrics-tls" not found
Sep 18 21:06:44.500571 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:44.500363    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls podName:abcc2145-3a92-486e-aaa8-16b8551f6bfe nodeName:}" failed. No retries permitted until 2026-09-18 21:06:52.500356959 +0000 UTC m=+50.107416494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls") pod "image-registry-84dd74ddb9-nbd9b" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe") : secret "image-registry-tls" not found
Sep 18 21:06:45.082609 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:45.082587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:45.159526 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:45.159501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:46.483125 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:46.483084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:48.039197 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:48.039164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:50.667233 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:50.667204    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:50.768955 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:50.768927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:50.852256 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:50.852228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:51.061520 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:51.061497    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:51.135303 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:51.135280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:51.314272 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:51.314206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:51.385296 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:51.385264    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:06:52.554022 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:52.553991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:06:52.554466 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:52.554037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:06:52.554466 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:52.554146    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:06:52.554466 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:52.554168    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84dd74ddb9-nbd9b: secret "image-registry-tls" not found
Sep 18 21:06:52.554466 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:52.554213    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:06:52.554466 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:52.554233    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls podName:abcc2145-3a92-486e-aaa8-16b8551f6bfe nodeName:}" failed. No retries permitted until 2026-09-18 21:07:08.554216146 +0000 UTC m=+66.161275691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls") pod "image-registry-84dd74ddb9-nbd9b" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe") : secret "image-registry-tls" not found
Sep 18 21:06:52.554466 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:06:52.554268    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls podName:d1225d31-f513-4cb7-b21a-a97accdb1bfb nodeName:}" failed. No retries permitted until 2026-09-18 21:07:08.554254762 +0000 UTC m=+66.161314299 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls") pod "dns-default-r9rc6" (UID: "d1225d31-f513-4cb7-b21a-a97accdb1bfb") : secret "dns-default-metrics-tls" not found
Sep 18 21:06:54.186109 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:06:54.186075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:02.220243 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:02.220211    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fpv5"
Sep 18 21:07:02.499919 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:02.499835    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:02.586474 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:02.586448    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:07.077946 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.077910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:07.179183 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.179155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:07.552738 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.552703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:07:07.557394 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.557374    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 21:07:07.567396 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.567368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2108fb-5c03-49d2-a864-2ef38a1fbaab-original-pull-secret\") pod \"global-pull-secret-syncer-rzd48\" (UID: \"5b2108fb-5c03-49d2-a864-2ef38a1fbaab\") " pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:07:07.627469 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.627443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzd48"
Sep 18 21:07:07.653661 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.653599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:07:07.656821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.656797    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 21:07:07.664100 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:07.664075    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 21:07:07.664218 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:07.664136    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs podName:0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa nodeName:}" failed. No retries permitted until 2026-09-18 21:08:11.664119543 +0000 UTC m=+129.271179078 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs") pod "network-metrics-daemon-bt5w7" (UID: "0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa") : secret "metrics-daemon-secret" not found
Sep 18 21:07:07.754764 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.754723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmngj\" (UniqueName: \"kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj\") pod \"network-check-target-pm2mb\" (UID: \"085474f0-4cdb-4b45-9fd2-f00a4e10af1c\") " pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:07:07.759999 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.759971    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 21:07:07.767709 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.767690    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 21:07:07.777997 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.777977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmngj\" (UniqueName: \"kubernetes.io/projected/085474f0-4cdb-4b45-9fd2-f00a4e10af1c-kube-api-access-mmngj\") pod \"network-check-target-pm2mb\" (UID: \"085474f0-4cdb-4b45-9fd2-f00a4e10af1c\") " pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:07:07.901417 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.901359    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rzd48"]
Sep 18 21:07:07.905998 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:07:07.905950    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2108fb_5c03_49d2_a864_2ef38a1fbaab.slice/crio-e516055c4208d1a1c6bec3879e5188ac5957a5f36531fca41d9c1bc56f87e118 WatchSource:0}: Error finding container e516055c4208d1a1c6bec3879e5188ac5957a5f36531fca41d9c1bc56f87e118: Status 404 returned error can't find the container with id e516055c4208d1a1c6bec3879e5188ac5957a5f36531fca41d9c1bc56f87e118
Sep 18 21:07:07.938008 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.937989    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xrfdl\""
Sep 18 21:07:07.945770 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:07.945748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:07:08.222856 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:08.222789    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pm2mb"]
Sep 18 21:07:08.225960 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:07:08.225930    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085474f0_4cdb_4b45_9fd2_f00a4e10af1c.slice/crio-b6afdd732081f3e577b495be690771f9d13eb615b48a311b2c5bf6be039af3d4 WatchSource:0}: Error finding container b6afdd732081f3e577b495be690771f9d13eb615b48a311b2c5bf6be039af3d4: Status 404 returned error can't find the container with id b6afdd732081f3e577b495be690771f9d13eb615b48a311b2c5bf6be039af3d4
Sep 18 21:07:08.312118 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:08.312090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pm2mb" event={"ID":"085474f0-4cdb-4b45-9fd2-f00a4e10af1c","Type":"ContainerStarted","Data":"b6afdd732081f3e577b495be690771f9d13eb615b48a311b2c5bf6be039af3d4"}
Sep 18 21:07:08.313331 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:08.313308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rzd48" event={"ID":"5b2108fb-5c03-49d2-a864-2ef38a1fbaab","Type":"ContainerStarted","Data":"e516055c4208d1a1c6bec3879e5188ac5957a5f36531fca41d9c1bc56f87e118"}
Sep 18 21:07:08.559203 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:08.559165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:07:08.559345 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:08.559219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:07:08.559345 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:08.559328    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:07:08.559461 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:08.559391    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls podName:d1225d31-f513-4cb7-b21a-a97accdb1bfb nodeName:}" failed. No retries permitted until 2026-09-18 21:07:40.559376944 +0000 UTC m=+98.166436480 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls") pod "dns-default-r9rc6" (UID: "d1225d31-f513-4cb7-b21a-a97accdb1bfb") : secret "dns-default-metrics-tls" not found
Sep 18 21:07:08.559461 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:08.559327    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 21:07:08.559461 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:08.559437    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84dd74ddb9-nbd9b: secret "image-registry-tls" not found
Sep 18 21:07:08.559620 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:08.559488    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls podName:abcc2145-3a92-486e-aaa8-16b8551f6bfe nodeName:}" failed. No retries permitted until 2026-09-18 21:07:40.559475175 +0000 UTC m=+98.166534711 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls") pod "image-registry-84dd74ddb9-nbd9b" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe") : secret "image-registry-tls" not found
Sep 18 21:07:12.696600 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:12.696569    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:12.772038 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:12.771975    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:13.327702 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:13.327670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pm2mb" event={"ID":"085474f0-4cdb-4b45-9fd2-f00a4e10af1c","Type":"ContainerStarted","Data":"7d29f8125d8360d2094e3e8e8f27a96e0aaf484408bbdb4b0a0cdc98065797c2"}
Sep 18 21:07:13.327966 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:13.327776    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:07:13.329386 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:13.329344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rzd48" event={"ID":"5b2108fb-5c03-49d2-a864-2ef38a1fbaab","Type":"ContainerStarted","Data":"57028c59f2f81b3fea969b0a6a00dff7b0a1efdb53c85cba8b934819d775f779"}
Sep 18 21:07:13.364239 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:13.364187    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pm2mb" podStartSLOduration=66.138607348 podStartE2EDuration="1m10.364172873s" totalImagesPullingTime="4.225565525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:13.363825175 +0000 UTC m=+70.970884736" watchObservedRunningTime="2026-09-18 21:07:13.364172873 +0000 UTC m=+70.971232510"
Sep 18 21:07:13.394737 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:13.394690    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rzd48" podStartSLOduration=65.843517946 podStartE2EDuration="1m10.394676637s" totalImagesPullingTime="4.551158691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:13.394602799 +0000 UTC m=+71.001662354" watchObservedRunningTime="2026-09-18 21:07:13.394676637 +0000 UTC m=+71.001736191"
Sep 18 21:07:16.426037 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.425991    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"]
Sep 18 21:07:16.429175 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.429146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"
Sep 18 21:07:16.433005 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.432976    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-jgkcm"]
Sep 18 21:07:16.433714 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.433636    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 21:07:16.433915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.433695    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xtwq6\""
Sep 18 21:07:16.433987 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.433958    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 21:07:16.434384 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.433773    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 21:07:16.434384 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.433806    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:07:16.438065 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.438044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.442119 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.442092    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"]
Sep 18 21:07:16.442246 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.442138    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 21:07:16.444662 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.444623    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bsddv\""
Sep 18 21:07:16.444817 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.444716    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 21:07:16.444946 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.444924    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"]
Sep 18 21:07:16.445029 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.445021    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"
Sep 18 21:07:16.445094 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.445057    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:07:16.445150 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.445116    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 21:07:16.450251 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.450226    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 21:07:16.450579 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.450555    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 21:07:16.450696 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.450619    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 21:07:16.450749 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.450698    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-525j6\""
Sep 18 21:07:16.450749 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.450703    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:07:16.464776 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.464754    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-jgkcm"]
Sep 18 21:07:16.465459 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.465439    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"]
Sep 18 21:07:16.516616 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.516595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1fbb0af0-328b-447e-824c-0d9d74840047-config\") pod \"console-operator-fd4bb46c-jgkcm\" (UID: \"1fbb0af0-328b-447e-824c-0d9d74840047\") " pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.516727 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.516635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p4wgq\" (UID: \"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"
Sep 18 21:07:16.516727 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.516664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47dg\" (UniqueName: \"kubernetes.io/projected/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-kube-api-access-h47dg\") pod \"cluster-samples-operator-685d7d68b5-p4wgq\" (UID: \"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"
Sep 18 21:07:16.516727 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.516703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc7pj\" (UniqueName: \"kubernetes.io/projected/9f5123e8-651a-474d-a808-38dc27a1063b-kube-api-access-gc7pj\") pod \"kube-storage-version-migrator-operator-55889f7b47-xmhzv\" (UID: \"9f5123e8-651a-474d-a808-38dc27a1063b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"
Sep 18 21:07:16.516895 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.516858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f5123e8-651a-474d-a808-38dc27a1063b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-xmhzv\" (UID: \"9f5123e8-651a-474d-a808-38dc27a1063b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"
Sep 18 21:07:16.516957 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.516923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6q4h\" (UniqueName: \"kubernetes.io/projected/1fbb0af0-328b-447e-824c-0d9d74840047-kube-api-access-n6q4h\") pod \"console-operator-fd4bb46c-jgkcm\" (UID: \"1fbb0af0-328b-447e-824c-0d9d74840047\") " pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.517006 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.516956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1fbb0af0-328b-447e-824c-0d9d74840047-serving-cert\") pod \"console-operator-fd4bb46c-jgkcm\" (UID: \"1fbb0af0-328b-447e-824c-0d9d74840047\") " pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.517006 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.516982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f5123e8-651a-474d-a808-38dc27a1063b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-xmhzv\" (UID: \"9f5123e8-651a-474d-a808-38dc27a1063b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"
Sep 18 21:07:16.517083 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.517005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fbb0af0-328b-447e-824c-0d9d74840047-trusted-ca\") pod \"console-operator-fd4bb46c-jgkcm\" (UID: \"1fbb0af0-328b-447e-824c-0d9d74840047\") " pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.618162 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.618138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fbb0af0-328b-447e-824c-0d9d74840047-trusted-ca\") pod \"console-operator-fd4bb46c-jgkcm\" (UID: \"1fbb0af0-328b-447e-824c-0d9d74840047\") " pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.618244 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.618167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1fbb0af0-328b-447e-824c-0d9d74840047-config\") pod \"console-operator-fd4bb46c-jgkcm\" (UID: \"1fbb0af0-328b-447e-824c-0d9d74840047\") " pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.618244 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.618196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p4wgq\" (UID: \"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"
Sep 18 21:07:16.618244 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.618213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47dg\" (UniqueName: \"kubernetes.io/projected/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-kube-api-access-h47dg\") pod \"cluster-samples-operator-685d7d68b5-p4wgq\" (UID: \"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"
Sep 18 21:07:16.618355 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.618257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc7pj\" (UniqueName: \"kubernetes.io/projected/9f5123e8-651a-474d-a808-38dc27a1063b-kube-api-access-gc7pj\") pod \"kube-storage-version-migrator-operator-55889f7b47-xmhzv\" (UID: \"9f5123e8-651a-474d-a808-38dc27a1063b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"
Sep 18 21:07:16.618355 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.618308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f5123e8-651a-474d-a808-38dc27a1063b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-xmhzv\" (UID: \"9f5123e8-651a-474d-a808-38dc27a1063b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"
Sep 18 21:07:16.618355 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:16.618313    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 21:07:16.618355 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.618333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6q4h\" (UniqueName: \"kubernetes.io/projected/1fbb0af0-328b-447e-824c-0d9d74840047-kube-api-access-n6q4h\") pod \"console-operator-fd4bb46c-jgkcm\" (UID: \"1fbb0af0-328b-447e-824c-0d9d74840047\") " pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.618542 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:16.618379    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls podName:e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1 nodeName:}" failed. No retries permitted until 2026-09-18 21:07:17.118361236 +0000 UTC m=+74.725420786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p4wgq" (UID: "e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1") : secret "samples-operator-tls" not found
Sep 18 21:07:16.618542 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.618396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1fbb0af0-328b-447e-824c-0d9d74840047-serving-cert\") pod \"console-operator-fd4bb46c-jgkcm\" (UID: \"1fbb0af0-328b-447e-824c-0d9d74840047\") " pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.618542 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.618426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f5123e8-651a-474d-a808-38dc27a1063b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-xmhzv\" (UID: \"9f5123e8-651a-474d-a808-38dc27a1063b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"
Sep 18 21:07:16.618977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.618945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f5123e8-651a-474d-a808-38dc27a1063b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-xmhzv\" (UID: \"9f5123e8-651a-474d-a808-38dc27a1063b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"
Sep 18 21:07:16.619077 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.618976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1fbb0af0-328b-447e-824c-0d9d74840047-config\") pod \"console-operator-fd4bb46c-jgkcm\" (UID: \"1fbb0af0-328b-447e-824c-0d9d74840047\") " pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.619124 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.619072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fbb0af0-328b-447e-824c-0d9d74840047-trusted-ca\") pod \"console-operator-fd4bb46c-jgkcm\" (UID: \"1fbb0af0-328b-447e-824c-0d9d74840047\") " pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.620834 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.620807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f5123e8-651a-474d-a808-38dc27a1063b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-xmhzv\" (UID: \"9f5123e8-651a-474d-a808-38dc27a1063b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"
Sep 18 21:07:16.620941 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.620853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1fbb0af0-328b-447e-824c-0d9d74840047-serving-cert\") pod \"console-operator-fd4bb46c-jgkcm\" (UID: \"1fbb0af0-328b-447e-824c-0d9d74840047\") " pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.627205 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.627184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc7pj\" (UniqueName: \"kubernetes.io/projected/9f5123e8-651a-474d-a808-38dc27a1063b-kube-api-access-gc7pj\") pod \"kube-storage-version-migrator-operator-55889f7b47-xmhzv\" (UID: \"9f5123e8-651a-474d-a808-38dc27a1063b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"
Sep 18 21:07:16.627451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.627430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6q4h\" (UniqueName: \"kubernetes.io/projected/1fbb0af0-328b-447e-824c-0d9d74840047-kube-api-access-n6q4h\") pod \"console-operator-fd4bb46c-jgkcm\" (UID: \"1fbb0af0-328b-447e-824c-0d9d74840047\") " pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.628773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.628749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47dg\" (UniqueName: \"kubernetes.io/projected/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-kube-api-access-h47dg\") pod \"cluster-samples-operator-685d7d68b5-p4wgq\" (UID: \"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"
Sep 18 21:07:16.713047 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.712991    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:16.745982 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.745958    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"
Sep 18 21:07:16.753055 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.753031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:16.782669 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.782608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:16.887756 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.887732    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:16.957615 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:16.957589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:17.021216 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:17.021166    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv"]
Sep 18 21:07:17.024483 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:17.024462    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-jgkcm"]
Sep 18 21:07:17.027240 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:07:17.027214    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5123e8_651a_474d_a808_38dc27a1063b.slice/crio-a1136e78e7a9048918c1ddd026924840c46a552d738987ae289755550626052e WatchSource:0}: Error finding container a1136e78e7a9048918c1ddd026924840c46a552d738987ae289755550626052e: Status 404 returned error can't find the container with id a1136e78e7a9048918c1ddd026924840c46a552d738987ae289755550626052e
Sep 18 21:07:17.028463 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:07:17.028444    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbb0af0_328b_447e_824c_0d9d74840047.slice/crio-340746aafbb7ab3636b89f09bdf41c771d97cd5df00892dd0a0732d880e161c0 WatchSource:0}: Error finding container 340746aafbb7ab3636b89f09bdf41c771d97cd5df00892dd0a0732d880e161c0: Status 404 returned error can't find the container with id 340746aafbb7ab3636b89f09bdf41c771d97cd5df00892dd0a0732d880e161c0
Sep 18 21:07:17.122059 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:17.122022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p4wgq\" (UID: \"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"
Sep 18 21:07:17.122177 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:17.122160    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 21:07:17.122223 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:17.122218    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls podName:e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1 nodeName:}" failed. No retries permitted until 2026-09-18 21:07:18.122199612 +0000 UTC m=+75.729259160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p4wgq" (UID: "e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1") : secret "samples-operator-tls" not found
Sep 18 21:07:17.340354 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:17.340322    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv" event={"ID":"9f5123e8-651a-474d-a808-38dc27a1063b","Type":"ContainerStarted","Data":"a1136e78e7a9048918c1ddd026924840c46a552d738987ae289755550626052e"}
Sep 18 21:07:17.341615 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:17.341588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm" event={"ID":"1fbb0af0-328b-447e-824c-0d9d74840047","Type":"ContainerStarted","Data":"340746aafbb7ab3636b89f09bdf41c771d97cd5df00892dd0a0732d880e161c0"}
Sep 18 21:07:18.130279 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:18.130245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p4wgq\" (UID: \"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"
Sep 18 21:07:18.130755 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:18.130413    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 21:07:18.130755 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:18.130494    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls podName:e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1 nodeName:}" failed. No retries permitted until 2026-09-18 21:07:20.130473236 +0000 UTC m=+77.737532796 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p4wgq" (UID: "e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1") : secret "samples-operator-tls" not found
Sep 18 21:07:20.146573 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:20.146545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p4wgq\" (UID: \"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"
Sep 18 21:07:20.146905 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:20.146693    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 21:07:20.146905 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:20.146756    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls podName:e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1 nodeName:}" failed. No retries permitted until 2026-09-18 21:07:24.146742894 +0000 UTC m=+81.753802430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p4wgq" (UID: "e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1") : secret "samples-operator-tls" not found
Sep 18 21:07:20.352291 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:20.352267    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/0.log"
Sep 18 21:07:20.352425 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:20.352303    2579 generic.go:356] "Generic (PLEG): container finished" podID="1fbb0af0-328b-447e-824c-0d9d74840047" containerID="5e31db197b5b73fa00f313a2cf8f2233487af8df38a3976bd8ec7d6629722e7f" exitCode=255
Sep 18 21:07:20.352425 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:20.352403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm" event={"ID":"1fbb0af0-328b-447e-824c-0d9d74840047","Type":"ContainerDied","Data":"5e31db197b5b73fa00f313a2cf8f2233487af8df38a3976bd8ec7d6629722e7f"}
Sep 18 21:07:20.352552 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:20.352537    2579 scope.go:117] "RemoveContainer" containerID="5e31db197b5b73fa00f313a2cf8f2233487af8df38a3976bd8ec7d6629722e7f"
Sep 18 21:07:20.353981 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:20.353958    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv" event={"ID":"9f5123e8-651a-474d-a808-38dc27a1063b","Type":"ContainerStarted","Data":"acd453cc26bb1b54ea0dbd2b8dcb5c9c0a3d375234a57560f01a4a58320ca62c"}
Sep 18 21:07:20.459237 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:20.459197    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv" podStartSLOduration=2.028753959 podStartE2EDuration="4.459182784s" totalImagesPullingTime="2.430428825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:20.45872427 +0000 UTC m=+78.065783830" watchObservedRunningTime="2026-09-18 21:07:20.459182784 +0000 UTC m=+78.066242342"
Sep 18 21:07:21.309734 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.309706    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:21.358818 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.358794    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:07:21.359177 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.359162    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/0.log"
Sep 18 21:07:21.359275 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.359198    2579 generic.go:356] "Generic (PLEG): container finished" podID="1fbb0af0-328b-447e-824c-0d9d74840047" containerID="64c087e90b6949d0dc882e692e0c3ef8f6790ab4a80ef5672b0b287b7e1acea9" exitCode=255
Sep 18 21:07:21.359336 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.359290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm" event={"ID":"1fbb0af0-328b-447e-824c-0d9d74840047","Type":"ContainerDied","Data":"64c087e90b6949d0dc882e692e0c3ef8f6790ab4a80ef5672b0b287b7e1acea9"}
Sep 18 21:07:21.359375 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.359355    2579 scope.go:117] "RemoveContainer" containerID="5e31db197b5b73fa00f313a2cf8f2233487af8df38a3976bd8ec7d6629722e7f"
Sep 18 21:07:21.359547 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.359533    2579 scope.go:117] "RemoveContainer" containerID="64c087e90b6949d0dc882e692e0c3ef8f6790ab4a80ef5672b0b287b7e1acea9"
Sep 18 21:07:21.359742 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:21.359718    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-jgkcm_openshift-console-operator(1fbb0af0-328b-447e-824c-0d9d74840047)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm" podUID="1fbb0af0-328b-447e-824c-0d9d74840047"
Sep 18 21:07:21.419939 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.419916    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-9cc7d"]
Sep 18 21:07:21.423860 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.423845    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9cc7d"
Sep 18 21:07:21.428856 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.428837    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 21:07:21.429304 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.429283    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wvjrr\""
Sep 18 21:07:21.430188 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.430127    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 21:07:21.436212 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.436189    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-9cc7d"]
Sep 18 21:07:21.455904 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.455880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl6s8\" (UniqueName: \"kubernetes.io/projected/003afec4-de4a-4ac0-98a1-8057c417f455-kube-api-access-zl6s8\") pod \"migrator-76794cbd6c-9cc7d\" (UID: \"003afec4-de4a-4ac0-98a1-8057c417f455\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9cc7d"
Sep 18 21:07:21.463406 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.463377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:21.556585 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.556560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6s8\" (UniqueName: \"kubernetes.io/projected/003afec4-de4a-4ac0-98a1-8057c417f455-kube-api-access-zl6s8\") pod \"migrator-76794cbd6c-9cc7d\" (UID: \"003afec4-de4a-4ac0-98a1-8057c417f455\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9cc7d"
Sep 18 21:07:21.567294 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.567245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl6s8\" (UniqueName: \"kubernetes.io/projected/003afec4-de4a-4ac0-98a1-8057c417f455-kube-api-access-zl6s8\") pod \"migrator-76794cbd6c-9cc7d\" (UID: \"003afec4-de4a-4ac0-98a1-8057c417f455\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9cc7d"
Sep 18 21:07:21.577331 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.577299    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:21.577754 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.577740    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j4x8n"]
Sep 18 21:07:21.580583 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.580568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:21.584421 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.584405    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 21:07:21.584744 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.584731    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lfhrj\""
Sep 18 21:07:21.585243 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.585224    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 21:07:21.585328 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.585248    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 21:07:21.585328 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.585287    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 21:07:21.600994 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.600966    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j4x8n"]
Sep 18 21:07:21.657191 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.657167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:21.657271 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.657200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82e6e1c8-fb55-4eff-b955-6376f98c29bc-data-volume\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:21.657271 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.657226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:21.657341 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.657292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82e6e1c8-fb55-4eff-b955-6376f98c29bc-crio-socket\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:21.657341 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.657315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75nsj\" (UniqueName: \"kubernetes.io/projected/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-api-access-75nsj\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:21.674068 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.674047    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:21.735399 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.735380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9cc7d"
Sep 18 21:07:21.758543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.758523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:21.758637 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.758553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82e6e1c8-fb55-4eff-b955-6376f98c29bc-data-volume\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:21.758637 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.758576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:21.758637 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.758600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82e6e1c8-fb55-4eff-b955-6376f98c29bc-crio-socket\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:21.758637 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.758629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75nsj\" (UniqueName: \"kubernetes.io/projected/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-api-access-75nsj\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:21.758854 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:21.758661    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:07:21.758854 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:21.758730    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm podName:82e6e1c8-fb55-4eff-b955-6376f98c29bc nodeName:}" failed. No retries permitted until 2026-09-18 21:07:22.258714046 +0000 UTC m=+79.865773582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j4x8n" (UID: "82e6e1c8-fb55-4eff-b955-6376f98c29bc") : configmap "kube-rbac-proxy" not found
Sep 18 21:07:21.758854 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:21.758734    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:07:21.758854 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.758766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82e6e1c8-fb55-4eff-b955-6376f98c29bc-crio-socket\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:21.758854 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:21.758796    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls podName:82e6e1c8-fb55-4eff-b955-6376f98c29bc nodeName:}" failed. No retries permitted until 2026-09-18 21:07:22.258778058 +0000 UTC m=+79.865837600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j4x8n" (UID: "82e6e1c8-fb55-4eff-b955-6376f98c29bc") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:07:21.759052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.758899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82e6e1c8-fb55-4eff-b955-6376f98c29bc-data-volume\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:21.770024 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:21.769998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75nsj\" (UniqueName: \"kubernetes.io/projected/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-api-access-75nsj\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:22.004461 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:22.004386    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-9cc7d"]
Sep 18 21:07:22.007504 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:07:22.007466    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003afec4_de4a_4ac0_98a1_8057c417f455.slice/crio-62bef19d1a2ccf50b4fea1e6a3f3aa7f5f01daa8fbe473b2ab0335f64a035af9 WatchSource:0}: Error finding container 62bef19d1a2ccf50b4fea1e6a3f3aa7f5f01daa8fbe473b2ab0335f64a035af9: Status 404 returned error can't find the container with id 62bef19d1a2ccf50b4fea1e6a3f3aa7f5f01daa8fbe473b2ab0335f64a035af9
Sep 18 21:07:22.263074 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:22.263010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:22.263074 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:22.263048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:22.263221 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:22.263133    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:07:22.263221 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:22.263153    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:07:22.263221 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:22.263201    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm podName:82e6e1c8-fb55-4eff-b955-6376f98c29bc nodeName:}" failed. No retries permitted until 2026-09-18 21:07:23.26318352 +0000 UTC m=+80.870243060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j4x8n" (UID: "82e6e1c8-fb55-4eff-b955-6376f98c29bc") : configmap "kube-rbac-proxy" not found
Sep 18 21:07:22.263221 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:22.263216    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls podName:82e6e1c8-fb55-4eff-b955-6376f98c29bc nodeName:}" failed. No retries permitted until 2026-09-18 21:07:23.263209657 +0000 UTC m=+80.870269192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j4x8n" (UID: "82e6e1c8-fb55-4eff-b955-6376f98c29bc") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:07:22.363277 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:22.363247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9cc7d" event={"ID":"003afec4-de4a-4ac0-98a1-8057c417f455","Type":"ContainerStarted","Data":"62bef19d1a2ccf50b4fea1e6a3f3aa7f5f01daa8fbe473b2ab0335f64a035af9"}
Sep 18 21:07:22.364878 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:22.364858    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:07:22.365165 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:22.365150    2579 scope.go:117] "RemoveContainer" containerID="64c087e90b6949d0dc882e692e0c3ef8f6790ab4a80ef5672b0b287b7e1acea9"
Sep 18 21:07:22.365310 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:22.365291    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-jgkcm_openshift-console-operator(1fbb0af0-328b-447e-824c-0d9d74840047)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm" podUID="1fbb0af0-328b-447e-824c-0d9d74840047"
Sep 18 21:07:23.150548 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:23.150523    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:23.250036 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:23.249985    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:23.272829 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:23.272804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:23.272894 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:23.272842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:23.272936 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:23.272917    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:07:23.272974 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:23.272959    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:07:23.273007 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:23.272979    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm podName:82e6e1c8-fb55-4eff-b955-6376f98c29bc nodeName:}" failed. No retries permitted until 2026-09-18 21:07:25.272965665 +0000 UTC m=+82.880025200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j4x8n" (UID: "82e6e1c8-fb55-4eff-b955-6376f98c29bc") : configmap "kube-rbac-proxy" not found
Sep 18 21:07:23.273380 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:23.273000    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls podName:82e6e1c8-fb55-4eff-b955-6376f98c29bc nodeName:}" failed. No retries permitted until 2026-09-18 21:07:25.272988165 +0000 UTC m=+82.880047700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j4x8n" (UID: "82e6e1c8-fb55-4eff-b955-6376f98c29bc") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:07:23.373428 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:23.373400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9cc7d" event={"ID":"003afec4-de4a-4ac0-98a1-8057c417f455","Type":"ContainerStarted","Data":"631cb6ae28b5e4c2e86ee6abfd537913ee8c375f2b031eeb58c7131c1c9f30da"}
Sep 18 21:07:23.373732 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:23.373482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9cc7d" event={"ID":"003afec4-de4a-4ac0-98a1-8057c417f455","Type":"ContainerStarted","Data":"7980fb7ee8b9ea344c3a495cef17c03735a59213c99ded691a5443664c407b44"}
Sep 18 21:07:23.407761 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:23.407719    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9cc7d" podStartSLOduration=1.418167724 podStartE2EDuration="2.407706131s" totalImagesPullingTime="989.538407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:23.406751196 +0000 UTC m=+81.013810753" watchObservedRunningTime="2026-09-18 21:07:23.407706131 +0000 UTC m=+81.014765685"
Sep 18 21:07:23.600119 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:23.600096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:23.703592 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:23.703568    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:23.875469 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:23.875416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:23.967235 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:23.967209    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:24.178937 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:24.178882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p4wgq\" (UID: \"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"
Sep 18 21:07:24.179074 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:24.179054    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 21:07:24.179149 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:24.179136    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls podName:e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1 nodeName:}" failed. No retries permitted until 2026-09-18 21:07:32.1791154 +0000 UTC m=+89.786174947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p4wgq" (UID: "e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1") : secret "samples-operator-tls" not found
Sep 18 21:07:24.521682 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:24.521600    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-khlzv_13cbaf64-d066-42f3-a262-8e56d161434a/dns-node-resolver/0.log"
Sep 18 21:07:25.288371 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:25.288268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:25.288371 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:25.288310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:25.288512 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:25.288406    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:07:25.288512 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:25.288485    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm podName:82e6e1c8-fb55-4eff-b955-6376f98c29bc nodeName:}" failed. No retries permitted until 2026-09-18 21:07:29.288466732 +0000 UTC m=+86.895526267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j4x8n" (UID: "82e6e1c8-fb55-4eff-b955-6376f98c29bc") : configmap "kube-rbac-proxy" not found
Sep 18 21:07:25.288512 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:25.288489    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:07:25.288610 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:25.288535    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls podName:82e6e1c8-fb55-4eff-b955-6376f98c29bc nodeName:}" failed. No retries permitted until 2026-09-18 21:07:29.288521713 +0000 UTC m=+86.895581251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j4x8n" (UID: "82e6e1c8-fb55-4eff-b955-6376f98c29bc") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:07:25.518732 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:25.518708    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tkw2z_dc1f208a-bfe4-49d1-9dac-1214ace566d8/node-ca/0.log"
Sep 18 21:07:25.522409 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:25.522373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:25.592419 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:25.592360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:25.921942 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:25.921890    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4brns_203754e1-73e8-415d-be7a-f187ea164375/serve-healthcheck-canary/0.log"
Sep 18 21:07:26.520618 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:26.520594    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-9cc7d_003afec4-de4a-4ac0-98a1-8057c417f455/migrator/0.log"
Sep 18 21:07:26.723869 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:26.723843    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-9cc7d_003afec4-de4a-4ac0-98a1-8057c417f455/graceful-termination/0.log"
Sep 18 21:07:26.754099 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:26.754074    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:26.754228 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:26.754117    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:26.754437 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:26.754423    2579 scope.go:117] "RemoveContainer" containerID="64c087e90b6949d0dc882e692e0c3ef8f6790ab4a80ef5672b0b287b7e1acea9"
Sep 18 21:07:26.754582 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:26.754565    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-jgkcm_openshift-console-operator(1fbb0af0-328b-447e-824c-0d9d74840047)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm" podUID="1fbb0af0-328b-447e-824c-0d9d74840047"
Sep 18 21:07:26.925958 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:26.925933    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-xmhzv_9f5123e8-651a-474d-a808-38dc27a1063b/kube-storage-version-migrator-operator/0.log"
Sep 18 21:07:27.005943 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:27.005911    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:27.133225 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:27.133195    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:27.319003 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:27.318973    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:27.427083 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:27.427056    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:27.722401 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:27.722343    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:28.516734 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:28.516700    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:28.691096 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:28.691074    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:29.175154 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:29.175124    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:29.316426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:29.316402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:29.316552 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:29.316439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:29.316552 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:29.316534    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:07:29.316552 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:29.316537    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:07:29.316676 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:29.316588    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls podName:82e6e1c8-fb55-4eff-b955-6376f98c29bc nodeName:}" failed. No retries permitted until 2026-09-18 21:07:37.31657407 +0000 UTC m=+94.923633607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j4x8n" (UID: "82e6e1c8-fb55-4eff-b955-6376f98c29bc") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:07:29.316676 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:07:29.316601    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm podName:82e6e1c8-fb55-4eff-b955-6376f98c29bc nodeName:}" failed. No retries permitted until 2026-09-18 21:07:37.316595286 +0000 UTC m=+94.923654822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j4x8n" (UID: "82e6e1c8-fb55-4eff-b955-6376f98c29bc") : configmap "kube-rbac-proxy" not found
Sep 18 21:07:29.701589 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:29.701553    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:30.085569 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:30.085541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:30.579816 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:30.579786    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:30.692505 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:30.692462    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:30.770859 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:30.770833    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:31.091295 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:31.091267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:31.192707 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:31.192677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:31.615531 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:31.615507    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:31.886159 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:31.886100    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:31.975379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:31.975353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:32.236261 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:32.236215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p4wgq\" (UID: \"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"
Sep 18 21:07:32.238277 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:32.238257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p4wgq\" (UID: \"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"
Sep 18 21:07:32.354916 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:32.354891    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:32.357771 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:32.357755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"
Sep 18 21:07:32.494849 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:32.494786    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:32.640451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:32.640425    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq"]
Sep 18 21:07:32.686414 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:32.686392    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:32.884570 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:32.884549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:33.409060 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:33.409022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq" event={"ID":"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1","Type":"ContainerStarted","Data":"66955ed2250b95a33b261388070a6074a1e1cf344a5a6a330dae77a25c374696"}
Sep 18 21:07:33.692418 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:33.692334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:34.086450 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:34.086410    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:34.318269 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:34.318234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:34.456156 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:34.456093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:34.556486 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:34.556460    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:34.758863 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:34.758796    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:35.094864 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:35.094825    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:35.419943 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:35.419876    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq" event={"ID":"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1","Type":"ContainerStarted","Data":"1fb3f4c178d33cf6edf8f822a9921bf309ee230120eede77923be56c0c457893"}
Sep 18 21:07:35.419943 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:35.419925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq" event={"ID":"e4a906db-5ba8-4e3d-a9f4-845e9b2b7dd1","Type":"ContainerStarted","Data":"b75e3642aedb00cd96a2e583da0faa1ea81ade9b916dc6883a9412a13ca4b568"}
Sep 18 21:07:35.436271 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:35.436220    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p4wgq" podStartSLOduration=17.637306832 podStartE2EDuration="19.436201257s" totalImagesPullingTime="1.798894425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:35.435565964 +0000 UTC m=+93.042625525" watchObservedRunningTime="2026-09-18 21:07:35.436201257 +0000 UTC m=+93.043260816"
Sep 18 21:07:35.879433 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:35.879399    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:36.286400 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:36.286356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:36.360281 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:36.360255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:36.429636 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:36.429611    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:36.509474 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:36.509447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:36.691244 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:36.691184    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:36.893611 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:36.893584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:37.374681 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:37.374636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:37.374784 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:37.374688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:37.375162 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:37.375142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6e1c8-fb55-4eff-b955-6376f98c29bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:37.377067 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:37.377044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6e1c8-fb55-4eff-b955-6376f98c29bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j4x8n\" (UID: \"82e6e1c8-fb55-4eff-b955-6376f98c29bc\") " pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:37.484949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:37.484920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:37.488817 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:37.488794    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j4x8n"
Sep 18 21:07:37.695319 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:37.695247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:37.760707 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:37.760684    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j4x8n"]
Sep 18 21:07:37.763105 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:07:37.763081    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e6e1c8_fb55_4eff_b955_6376f98c29bc.slice/crio-1e8616220a076f2aa0b867fb01f83e629464db518a189d1b24d5faba5bfb1ca4 WatchSource:0}: Error finding container 1e8616220a076f2aa0b867fb01f83e629464db518a189d1b24d5faba5bfb1ca4: Status 404 returned error can't find the container with id 1e8616220a076f2aa0b867fb01f83e629464db518a189d1b24d5faba5bfb1ca4
Sep 18 21:07:37.916527 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:37.916507    2579 scope.go:117] "RemoveContainer" containerID="64c087e90b6949d0dc882e692e0c3ef8f6790ab4a80ef5672b0b287b7e1acea9"
Sep 18 21:07:38.432968 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:38.432939    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:07:38.433141 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:38.433056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm" event={"ID":"1fbb0af0-328b-447e-824c-0d9d74840047","Type":"ContainerStarted","Data":"5640be05a223f7829fb7519d6b94d02a22c252239ec739bf0ffbdebb693a624c"}
Sep 18 21:07:38.433442 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:38.433326    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:38.434684 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:38.434654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j4x8n" event={"ID":"82e6e1c8-fb55-4eff-b955-6376f98c29bc","Type":"ContainerStarted","Data":"0a105f4f0a62c22b2a8954b1d7779917fb83fd6ffb4be97f6cd0b75fa95b9eae"}
Sep 18 21:07:38.434783 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:38.434713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j4x8n" event={"ID":"82e6e1c8-fb55-4eff-b955-6376f98c29bc","Type":"ContainerStarted","Data":"1e8616220a076f2aa0b867fb01f83e629464db518a189d1b24d5faba5bfb1ca4"}
Sep 18 21:07:38.717998 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:38.717946    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm"
Sep 18 21:07:38.737099 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:38.737049    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-jgkcm" podStartSLOduration=20.309856598 podStartE2EDuration="22.737029815s" totalImagesPullingTime="2.427173217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:38.454272095 +0000 UTC m=+96.061331652" watchObservedRunningTime="2026-09-18 21:07:38.737029815 +0000 UTC m=+96.344089375"
Sep 18 21:07:38.742444 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:38.742417    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:38.953320 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:38.953279    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:39.027795 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:39.027729    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:39.217481 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:39.217452    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:39.384234 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:39.384208    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:39.441531 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:39.441499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j4x8n" event={"ID":"82e6e1c8-fb55-4eff-b955-6376f98c29bc","Type":"ContainerStarted","Data":"98e044479f5b3306ae75b6fee88391df2482693aa98aa714ffcae1063cfd77ab"}
Sep 18 21:07:39.448135 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:39.448108    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:39.578905 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:39.578875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:39.645714 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:39.645634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:39.787174 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:39.787144    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:39.862564 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:39.862539    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:40.447399 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.447366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j4x8n" event={"ID":"82e6e1c8-fb55-4eff-b955-6376f98c29bc","Type":"ContainerStarted","Data":"160c93f0513f5ef34ecae83e059fc39fa640942ef197e0ab596f88c21a99312e"}
Sep 18 21:07:40.484870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.484825    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j4x8n" podStartSLOduration=17.482344245 podStartE2EDuration="19.484808681s" totalImagesPullingTime="2.002464436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:07:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:40.483931772 +0000 UTC m=+98.090991334" watchObservedRunningTime="2026-09-18 21:07:40.484808681 +0000 UTC m=+98.091868239"
Sep 18 21:07:40.598506 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.598476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:40.599508 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.599486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:07:40.599671 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.599630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:07:40.601761 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.601740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls\") pod \"image-registry-84dd74ddb9-nbd9b\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") " pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:07:40.601761 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.601745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1225d31-f513-4cb7-b21a-a97accdb1bfb-metrics-tls\") pod \"dns-default-r9rc6\" (UID: \"d1225d31-f513-4cb7-b21a-a97accdb1bfb\") " pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:07:40.645472 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.645453    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4xcf\""
Sep 18 21:07:40.653765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.653750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:07:40.792083 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.792055    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:40.885062 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.885032    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d7nd7\""
Sep 18 21:07:40.893366 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.893342    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:07:40.922011 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.921986    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9rc6"]
Sep 18 21:07:40.924875 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:07:40.924852    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1225d31_f513_4cb7_b21a_a97accdb1bfb.slice/crio-b280a9576d0e341140c49a495bcfe0e9ecd21043c60642c4b559e5907eb7669d WatchSource:0}: Error finding container b280a9576d0e341140c49a495bcfe0e9ecd21043c60642c4b559e5907eb7669d: Status 404 returned error can't find the container with id b280a9576d0e341140c49a495bcfe0e9ecd21043c60642c4b559e5907eb7669d
Sep 18 21:07:40.998370 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:40.998318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:41.174352 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:41.174207    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84dd74ddb9-nbd9b"]
Sep 18 21:07:41.176920 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:07:41.176892    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabcc2145_3a92_486e_aaa8_16b8551f6bfe.slice/crio-d2af6fdb04e1a01e03e38d1ef5318d8eecba523422f847de36198bbc83a5337d WatchSource:0}: Error finding container d2af6fdb04e1a01e03e38d1ef5318d8eecba523422f847de36198bbc83a5337d: Status 404 returned error can't find the container with id d2af6fdb04e1a01e03e38d1ef5318d8eecba523422f847de36198bbc83a5337d
Sep 18 21:07:41.407144 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:41.407092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:41.452051 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:41.452015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b" event={"ID":"abcc2145-3a92-486e-aaa8-16b8551f6bfe","Type":"ContainerStarted","Data":"a5a61d4c4c5c0f08734c77e9df0a1ae8bd2cd59c9c392cef1e4593ac13820c91"}
Sep 18 21:07:41.452464 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:41.452068    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b" event={"ID":"abcc2145-3a92-486e-aaa8-16b8551f6bfe","Type":"ContainerStarted","Data":"d2af6fdb04e1a01e03e38d1ef5318d8eecba523422f847de36198bbc83a5337d"}
Sep 18 21:07:41.452464 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:41.452128    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:07:41.453240 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:41.453214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9rc6" event={"ID":"d1225d31-f513-4cb7-b21a-a97accdb1bfb","Type":"ContainerStarted","Data":"b280a9576d0e341140c49a495bcfe0e9ecd21043c60642c4b559e5907eb7669d"}
Sep 18 21:07:41.475835 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:41.475790    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b" podStartSLOduration=98.475777615 podStartE2EDuration="1m38.475777615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:41.474719856 +0000 UTC m=+99.081779414" watchObservedRunningTime="2026-09-18 21:07:41.475777615 +0000 UTC m=+99.082837170"
Sep 18 21:07:43.462014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:43.461979    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9rc6" event={"ID":"d1225d31-f513-4cb7-b21a-a97accdb1bfb","Type":"ContainerStarted","Data":"d431f02c1daa7f25d4cd3b6666d8b7a2c5efd1d8dd7ff631888c91f1d3e06473"}
Sep 18 21:07:43.462368 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:43.462032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9rc6" event={"ID":"d1225d31-f513-4cb7-b21a-a97accdb1bfb","Type":"ContainerStarted","Data":"f257372cc398f629fabf16a6db69563cdb18c803777f5a86b7175310f2a28888"}
Sep 18 21:07:43.462368 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:43.462133    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:07:43.485071 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:43.485019    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r9rc6" podStartSLOduration=65.483457552 podStartE2EDuration="1m7.485001647s" totalImagesPullingTime="2.001544095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:07:43.484227983 +0000 UTC m=+101.091287543" watchObservedRunningTime="2026-09-18 21:07:43.485001647 +0000 UTC m=+101.092061205"
Sep 18 21:07:44.334176 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:44.334140    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pm2mb"
Sep 18 21:07:47.632814 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:47.632782    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-pdrnp"]
Sep 18 21:07:47.635903 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:47.635884    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-pdrnp"
Sep 18 21:07:47.644821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:47.644803    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fj2ds\""
Sep 18 21:07:47.644969 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:47.644943    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 21:07:47.645794 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:47.645778    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 21:07:47.694941 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:47.694923    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-pdrnp"]
Sep 18 21:07:47.743616 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:47.743594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-798rz\" (UniqueName: \"kubernetes.io/projected/22b37c6f-d508-4c3e-a969-238c8900482c-kube-api-access-798rz\") pod \"downloads-6594f4c75-pdrnp\" (UID: \"22b37c6f-d508-4c3e-a969-238c8900482c\") " pod="openshift-console/downloads-6594f4c75-pdrnp"
Sep 18 21:07:47.844754 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:47.844730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-798rz\" (UniqueName: \"kubernetes.io/projected/22b37c6f-d508-4c3e-a969-238c8900482c-kube-api-access-798rz\") pod \"downloads-6594f4c75-pdrnp\" (UID: \"22b37c6f-d508-4c3e-a969-238c8900482c\") " pod="openshift-console/downloads-6594f4c75-pdrnp"
Sep 18 21:07:47.881701 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:47.881671    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:47.884988 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:47.884937    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-798rz\" (UniqueName: \"kubernetes.io/projected/22b37c6f-d508-4c3e-a969-238c8900482c-kube-api-access-798rz\") pod \"downloads-6594f4c75-pdrnp\" (UID: \"22b37c6f-d508-4c3e-a969-238c8900482c\") " pod="openshift-console/downloads-6594f4c75-pdrnp"
Sep 18 21:07:47.944029 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:47.944001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-pdrnp"
Sep 18 21:07:48.017300 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:48.017277    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:48.226344 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:48.226288    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-pdrnp"]
Sep 18 21:07:48.229553 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:07:48.229522    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b37c6f_d508_4c3e_a969_238c8900482c.slice/crio-d39da0e57fbf04007100b99b8c8a82c5eb0fc1895fbe5ce4b886dc0cc6668f59 WatchSource:0}: Error finding container d39da0e57fbf04007100b99b8c8a82c5eb0fc1895fbe5ce4b886dc0cc6668f59: Status 404 returned error can't find the container with id d39da0e57fbf04007100b99b8c8a82c5eb0fc1895fbe5ce4b886dc0cc6668f59
Sep 18 21:07:48.481828 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:48.481755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-pdrnp" event={"ID":"22b37c6f-d508-4c3e-a969-238c8900482c","Type":"ContainerStarted","Data":"d39da0e57fbf04007100b99b8c8a82c5eb0fc1895fbe5ce4b886dc0cc6668f59"}
Sep 18 21:07:49.975828 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:49.975790    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:50.100827 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:50.100793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:52.248894 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.248860    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6747499786-fwmb4"]
Sep 18 21:07:52.251324 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.251299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.254443 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.254422    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 21:07:52.254631 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.254609    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 21:07:52.254749 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.254668    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 21:07:52.255636 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.255617    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 21:07:52.255748 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.255725    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9pwwm\""
Sep 18 21:07:52.255812 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.255790    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 21:07:52.272224 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.272199    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6747499786-fwmb4"]
Sep 18 21:07:52.377986 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.377960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-serving-cert\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.378129 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.378015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ztmq\" (UniqueName: \"kubernetes.io/projected/fb270b78-adc7-4082-9fd9-4b789ad7aa65-kube-api-access-2ztmq\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.378129 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.378045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-config\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.378129 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.378066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-oauth-config\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.378129 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.378120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-service-ca\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.378328 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.378146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-oauth-serving-cert\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.479375 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.479346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-serving-cert\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.479504 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.479397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ztmq\" (UniqueName: \"kubernetes.io/projected/fb270b78-adc7-4082-9fd9-4b789ad7aa65-kube-api-access-2ztmq\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.479504 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.479427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-config\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.479504 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.479452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-oauth-config\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.479504 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.479487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-service-ca\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.479723 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.479516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-oauth-serving-cert\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.480471 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.480421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-config\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.480471 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.480438    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-oauth-serving-cert\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.480632 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.480500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-service-ca\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.482473 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.482452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-oauth-config\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.482565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.482454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-serving-cert\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.489113 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.489090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ztmq\" (UniqueName: \"kubernetes.io/projected/fb270b78-adc7-4082-9fd9-4b789ad7aa65-kube-api-access-2ztmq\") pod \"console-6747499786-fwmb4\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") " pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.562011 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.561985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:07:52.738385 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.738324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:52.843907 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.843789    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:52.861810 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.861791    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6747499786-fwmb4"]
Sep 18 21:07:52.864603 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:07:52.864569    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb270b78_adc7_4082_9fd9_4b789ad7aa65.slice/crio-b8a9948e26c385dee200f029ea97033a96b18f1b4bacd469bc9137498f73c5bd WatchSource:0}: Error finding container b8a9948e26c385dee200f029ea97033a96b18f1b4bacd469bc9137498f73c5bd: Status 404 returned error can't find the container with id b8a9948e26c385dee200f029ea97033a96b18f1b4bacd469bc9137498f73c5bd
Sep 18 21:07:52.938221 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.938193    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-mqvzc"]
Sep 18 21:07:52.943234 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.943202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:52.956966 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.956938    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-mqvzc"]
Sep 18 21:07:52.957088 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.957062    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-mqvzc"
Sep 18 21:07:52.960628 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.960605    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fxmh7\""
Sep 18 21:07:52.960735 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:52.960605    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 21:07:53.066821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:53.066792    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:53.084682 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:53.084633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bfee6987-d86e-440b-b097-0e4b5539d8c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-mqvzc\" (UID: \"bfee6987-d86e-440b-b097-0e4b5539d8c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-mqvzc"
Sep 18 21:07:53.185577 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:53.185521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bfee6987-d86e-440b-b097-0e4b5539d8c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-mqvzc\" (UID: \"bfee6987-d86e-440b-b097-0e4b5539d8c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-mqvzc"
Sep 18 21:07:53.188141 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:53.188122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bfee6987-d86e-440b-b097-0e4b5539d8c4-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-mqvzc\" (UID: \"bfee6987-d86e-440b-b097-0e4b5539d8c4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-mqvzc"
Sep 18 21:07:53.268001 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:53.267979    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-mqvzc"
Sep 18 21:07:53.467913 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:53.467731    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r9rc6"
Sep 18 21:07:53.507544 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:53.507505    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6747499786-fwmb4" event={"ID":"fb270b78-adc7-4082-9fd9-4b789ad7aa65","Type":"ContainerStarted","Data":"b8a9948e26c385dee200f029ea97033a96b18f1b4bacd469bc9137498f73c5bd"}
Sep 18 21:07:53.574056 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:53.573982    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-mqvzc"]
Sep 18 21:07:53.586503 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:07:53.586465    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfee6987_d86e_440b_b097_0e4b5539d8c4.slice/crio-abafe3e7d97a826aac78a1f87339e5e26758826dca662557fef127b62f44fc1d WatchSource:0}: Error finding container abafe3e7d97a826aac78a1f87339e5e26758826dca662557fef127b62f44fc1d: Status 404 returned error can't find the container with id abafe3e7d97a826aac78a1f87339e5e26758826dca662557fef127b62f44fc1d
Sep 18 21:07:54.021037 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:54.020983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:54.121486 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:54.121400    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:54.514612 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:54.514542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-mqvzc" event={"ID":"bfee6987-d86e-440b-b097-0e4b5539d8c4","Type":"ContainerStarted","Data":"abafe3e7d97a826aac78a1f87339e5e26758826dca662557fef127b62f44fc1d"}
Sep 18 21:07:56.063378 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:56.063326    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:56.160109 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:56.160062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:57.863079 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:57.863046    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:57.937860 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:57.937824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:59.218443 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:59.218410    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:07:59.319202 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:07:59.319166    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:00.899998 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:00.899958    2579 patch_prober.go:28] interesting pod/image-registry-84dd74ddb9-nbd9b container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 21:08:00.900406 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:00.900038    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b" podUID="abcc2145-3a92-486e-aaa8-16b8551f6bfe" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 21:08:01.848909 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:01.848875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:01.950473 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:01.950437    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:02.461946 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:02.461905    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:08:02.733752 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:02.733673    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:02.827299 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:02.827263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:05.557434 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:05.557392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6747499786-fwmb4" event={"ID":"fb270b78-adc7-4082-9fd9-4b789ad7aa65","Type":"ContainerStarted","Data":"f583825893a38c5d52c39cd623ccd86a18e88a88c1f7f202a756d2c09947862b"}
Sep 18 21:08:05.559439 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:05.559385    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-mqvzc" event={"ID":"bfee6987-d86e-440b-b097-0e4b5539d8c4","Type":"ContainerStarted","Data":"91e4b31e857f2e65ee99b8ee7b37a360f6e2dafd9c7a81c498b94c5e6580b29b"}
Sep 18 21:08:05.559579 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:05.559561    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-mqvzc"
Sep 18 21:08:05.561321 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:05.561299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-pdrnp" event={"ID":"22b37c6f-d508-4c3e-a969-238c8900482c","Type":"ContainerStarted","Data":"9483a62bc96741f95f97dde88fa64d02199aa0cabab8beecc9e5c65569a9b769"}
Sep 18 21:08:05.561550 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:05.561526    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-pdrnp"
Sep 18 21:08:05.565277 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:05.565255    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-mqvzc"
Sep 18 21:08:05.576532 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:05.576514    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-pdrnp"
Sep 18 21:08:05.585567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:05.585514    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6747499786-fwmb4" podStartSLOduration=1.807181297 podStartE2EDuration="13.585496596s" totalImagesPullingTime="11.778315299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:05.579983122 +0000 UTC m=+123.187042674" watchObservedRunningTime="2026-09-18 21:08:05.585496596 +0000 UTC m=+123.192556198"
Sep 18 21:08:05.611654 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:05.611599    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-pdrnp" podStartSLOduration=2.160931228 podStartE2EDuration="18.611581237s" totalImagesPullingTime="16.450650009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:05.611269842 +0000 UTC m=+123.218329400" watchObservedRunningTime="2026-09-18 21:08:05.611581237 +0000 UTC m=+123.218640794"
Sep 18 21:08:05.637692 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:05.637631    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-mqvzc" podStartSLOduration=2.598577399 podStartE2EDuration="13.637617626s" totalImagesPullingTime="11.039040227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:05.636919635 +0000 UTC m=+123.243979191" watchObservedRunningTime="2026-09-18 21:08:05.637617626 +0000 UTC m=+123.244677184"
Sep 18 21:08:05.998930 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:05.998852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:06.147988 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:06.147947    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:06.304887 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:06.304851    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:06.424532 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:06.424495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:07.317498 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:07.317458    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:07.626445 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:07.626344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:07.865952 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:07.865898    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:08.710983 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:08.710928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:09.788813 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:09.788776    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84dd74ddb9-nbd9b"]
Sep 18 21:08:09.883470 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:09.883426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:10.007065 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.007031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:10.120471 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.120404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:10.260564 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.260531    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:10.485533 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.485459    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:10.552502 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.552468    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-scjlc"]
Sep 18 21:08:10.582898 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.582868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.585495 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.585464    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 21:08:10.585795 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.585771    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 21:08:10.585938 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.585893    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 21:08:10.586012 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.585998    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 21:08:10.586077 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.585999    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 21:08:10.586077 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.586019    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-lh95h"]
Sep 18 21:08:10.586825 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.586803    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 21:08:10.586936 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.586831    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cktl2\""
Sep 18 21:08:10.612866 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.612846    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-lh95h"]
Sep 18 21:08:10.612995 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.612952    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.615665 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.615622    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 21:08:10.616183 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.615989    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 21:08:10.616183 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.616027    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 21:08:10.616183 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.615989    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-c5lj7\""
Sep 18 21:08:10.620150 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.620114    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:10.718201 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.718169    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:10.724183 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.724305 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/153a9d01-c1ff-471d-a43e-9387dc4917e5-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.724305 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbqf\" (UniqueName: \"kubernetes.io/projected/153a9d01-c1ff-471d-a43e-9387dc4917e5-kube-api-access-vlbqf\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.724305 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/153a9d01-c1ff-471d-a43e-9387dc4917e5-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.724305 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-accelerators-collector-config\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.724305 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d27c900a-494e-4028-b6bc-ed4794f33e63-root\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.724549 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-wtmp\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.724549 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d27c900a-494e-4028-b6bc-ed4794f33e63-metrics-client-ca\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.724549 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d27c900a-494e-4028-b6bc-ed4794f33e63-sys\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.724549 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/153a9d01-c1ff-471d-a43e-9387dc4917e5-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.724740 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-textfile\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.724740 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssq65\" (UniqueName: \"kubernetes.io/projected/d27c900a-494e-4028-b6bc-ed4794f33e63-kube-api-access-ssq65\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.724740 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/153a9d01-c1ff-471d-a43e-9387dc4917e5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.724740 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-tls\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.724896 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.724744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/153a9d01-c1ff-471d-a43e-9387dc4917e5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.825177 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-accelerators-collector-config\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d27c900a-494e-4028-b6bc-ed4794f33e63-root\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-wtmp\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d27c900a-494e-4028-b6bc-ed4794f33e63-metrics-client-ca\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d27c900a-494e-4028-b6bc-ed4794f33e63-root\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d27c900a-494e-4028-b6bc-ed4794f33e63-sys\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825328    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d27c900a-494e-4028-b6bc-ed4794f33e63-sys\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/153a9d01-c1ff-471d-a43e-9387dc4917e5-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-textfile\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssq65\" (UniqueName: \"kubernetes.io/projected/d27c900a-494e-4028-b6bc-ed4794f33e63-kube-api-access-ssq65\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/153a9d01-c1ff-471d-a43e-9387dc4917e5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-wtmp\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-tls\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/153a9d01-c1ff-471d-a43e-9387dc4917e5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.825543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.826329 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/153a9d01-c1ff-471d-a43e-9387dc4917e5-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.826329 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbqf\" (UniqueName: \"kubernetes.io/projected/153a9d01-c1ff-471d-a43e-9387dc4917e5-kube-api-access-vlbqf\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.826329 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/153a9d01-c1ff-471d-a43e-9387dc4917e5-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.826329 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/153a9d01-c1ff-471d-a43e-9387dc4917e5-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.826329 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-accelerators-collector-config\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.826329 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.825949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d27c900a-494e-4028-b6bc-ed4794f33e63-metrics-client-ca\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.826625 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.826340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/153a9d01-c1ff-471d-a43e-9387dc4917e5-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.826625 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.826465    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/153a9d01-c1ff-471d-a43e-9387dc4917e5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.826758 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.826728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-textfile\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.828434 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.828383    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-tls\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.828530 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.828450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/153a9d01-c1ff-471d-a43e-9387dc4917e5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.828736 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.828714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d27c900a-494e-4028-b6bc-ed4794f33e63-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.828866 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.828839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/153a9d01-c1ff-471d-a43e-9387dc4917e5-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.833753 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.833732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssq65\" (UniqueName: \"kubernetes.io/projected/d27c900a-494e-4028-b6bc-ed4794f33e63-kube-api-access-ssq65\") pod \"node-exporter-scjlc\" (UID: \"d27c900a-494e-4028-b6bc-ed4794f33e63\") " pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.834929 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.834907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbqf\" (UniqueName: \"kubernetes.io/projected/153a9d01-c1ff-471d-a43e-9387dc4917e5-kube-api-access-vlbqf\") pod \"kube-state-metrics-84c885658c-lh95h\" (UID: \"153a9d01-c1ff-471d-a43e-9387dc4917e5\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.868465 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.868439    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6747499786-fwmb4"]
Sep 18 21:08:10.894533 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.894510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-scjlc"
Sep 18 21:08:10.925005 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.924981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h"
Sep 18 21:08:10.935247 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:08:10.935220    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27c900a_494e_4028_b6bc_ed4794f33e63.slice/crio-5c0ef7f6b8188d33918862d1e39ac381ba21b16f1e7b8e0b74c153c5eff3f8c0 WatchSource:0}: Error finding container 5c0ef7f6b8188d33918862d1e39ac381ba21b16f1e7b8e0b74c153c5eff3f8c0: Status 404 returned error can't find the container with id 5c0ef7f6b8188d33918862d1e39ac381ba21b16f1e7b8e0b74c153c5eff3f8c0
Sep 18 21:08:10.954794 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:10.954768    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:11.050713 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.050369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:11.147620 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.147525    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:11.244543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.244337    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-lh95h"]
Sep 18 21:08:11.586551 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.586507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h" event={"ID":"153a9d01-c1ff-471d-a43e-9387dc4917e5","Type":"ContainerStarted","Data":"d02a9a741838306455c7d1015c2afb45dd6122b8c25c8f4ed11c1742ca213344"}
Sep 18 21:08:11.588368 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.588323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-scjlc" event={"ID":"d27c900a-494e-4028-b6bc-ed4794f33e63","Type":"ContainerStarted","Data":"5c0ef7f6b8188d33918862d1e39ac381ba21b16f1e7b8e0b74c153c5eff3f8c0"}
Sep 18 21:08:11.620603 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.620566    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:11.724215 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.724180    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:11.736720 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.736684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:08:11.739275 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.739253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa-metrics-certs\") pod \"network-metrics-daemon-bt5w7\" (UID: \"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa\") " pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:08:11.750606 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.750580    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 21:08:11.771556 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.771533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:11.774968 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.774915    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 21:08:11.774968 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.774940    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 21:08:11.775133 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.775100    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 21:08:11.775635 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.775613    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 21:08:11.775974 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.775783    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 21:08:11.775974 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.775878    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 21:08:11.775974 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.775907    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 21:08:11.775974 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.775937    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wrhgb\""
Sep 18 21:08:11.776238 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.776043    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 21:08:11.776694 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.776676    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 21:08:11.787889 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.787822    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 21:08:11.843203 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.843139    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w6vrc\""
Sep 18 21:08:11.851547 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.851525    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bt5w7"
Sep 18 21:08:11.895591 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.895529    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:11.938868 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.938840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:11.938978 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.938905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:11.938978 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.938938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:11.939093 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.938979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-config-out\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:11.939093 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.939006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:11.939093 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.939031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:11.939093 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.939056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:11.939093 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.939089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:11.939303 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.939114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:11.939303 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.939143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:11.939303 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.939163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:11.939303 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.939192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzfwt\" (UniqueName: \"kubernetes.io/projected/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-kube-api-access-pzfwt\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:11.939303 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:11.939222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-web-config\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.039689 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.039637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.039827 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.039732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.039827 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.039765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.039943 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.039832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-config-out\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.039943 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.039858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.039943 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.039882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.039943 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.039908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.040148 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.039944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.040148 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.039969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.040148 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.040004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.040148 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.040037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.040148 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.040072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzfwt\" (UniqueName: \"kubernetes.io/projected/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-kube-api-access-pzfwt\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.040148 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.040101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-web-config\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.041089 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.040736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.041089 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.040797    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.042849 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.042821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.044052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.043736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.044052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.043808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.044708 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.044239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.045480 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.045056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-config-out\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.045480 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.045132    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.045480 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.045326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.045480 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.045470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.046540 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.046515    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:12.046663 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.046543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.047769 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.047731    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-web-config\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.049688 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.049666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzfwt\" (UniqueName: \"kubernetes.io/projected/3ae6fa83-229b-4b0c-b6fd-4d39f54041fd-kube-api-access-pzfwt\") pod \"alertmanager-main-0\" (UID: \"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.083602 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.083578    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 21:08:12.399061 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.399030    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bt5w7"]
Sep 18 21:08:12.412516 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.412484    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 21:08:12.424108 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:08:12.424072    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddfe7f6_f27f_4791_a7da_31a0c34fa3fa.slice/crio-c1700edc5f2f93129ac992ce35c37ebaa9e57c4c12aa4d9e2c18746729f0cad3 WatchSource:0}: Error finding container c1700edc5f2f93129ac992ce35c37ebaa9e57c4c12aa4d9e2c18746729f0cad3: Status 404 returned error can't find the container with id c1700edc5f2f93129ac992ce35c37ebaa9e57c4c12aa4d9e2c18746729f0cad3
Sep 18 21:08:12.425218 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:08:12.425174    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae6fa83_229b_4b0c_b6fd_4d39f54041fd.slice/crio-476862090f528613aac9210d81b2bc73be3be41f1dbe72c333463e2646e05d8c WatchSource:0}: Error finding container 476862090f528613aac9210d81b2bc73be3be41f1dbe72c333463e2646e05d8c: Status 404 returned error can't find the container with id 476862090f528613aac9210d81b2bc73be3be41f1dbe72c333463e2646e05d8c
Sep 18 21:08:12.563007 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.562980    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:08:12.594392 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.594309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd","Type":"ContainerStarted","Data":"476862090f528613aac9210d81b2bc73be3be41f1dbe72c333463e2646e05d8c"}
Sep 18 21:08:12.596440 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.596376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bt5w7" event={"ID":"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa","Type":"ContainerStarted","Data":"c1700edc5f2f93129ac992ce35c37ebaa9e57c4c12aa4d9e2c18746729f0cad3"}
Sep 18 21:08:12.598558 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:12.598528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-scjlc" event={"ID":"d27c900a-494e-4028-b6bc-ed4794f33e63","Type":"ContainerStarted","Data":"44ddd09ae356dfcc26a512e582545ab40fe3e432da5df395f06e0483c5d1c598"}
Sep 18 21:08:13.595514 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:13.595466    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:13.606154 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:13.606021    2579 generic.go:356] "Generic (PLEG): container finished" podID="d27c900a-494e-4028-b6bc-ed4794f33e63" containerID="44ddd09ae356dfcc26a512e582545ab40fe3e432da5df395f06e0483c5d1c598" exitCode=0
Sep 18 21:08:13.606154 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:13.606101    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-scjlc" event={"ID":"d27c900a-494e-4028-b6bc-ed4794f33e63","Type":"ContainerDied","Data":"44ddd09ae356dfcc26a512e582545ab40fe3e432da5df395f06e0483c5d1c598"}
Sep 18 21:08:13.734394 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:13.734299    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:14.011716 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:14.011612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:15.121404 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.121377    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6467685b8-xgnwr"]
Sep 18 21:08:15.144693 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.144635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.148914 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.148894    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 21:08:15.149024 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.148949    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 21:08:15.150014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.149994    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-f9wcp\""
Sep 18 21:08:15.150154 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.150121    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 21:08:15.150154 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.150150    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4qb5dq3mhh5jc\""
Sep 18 21:08:15.150485 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.150434    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 21:08:15.152155 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.152105    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6467685b8-xgnwr"]
Sep 18 21:08:15.170115 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.170090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/329862e1-1940-4da0-9cca-483aec6e3486-client-ca-bundle\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.170257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.170139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npg4p\" (UniqueName: \"kubernetes.io/projected/329862e1-1940-4da0-9cca-483aec6e3486-kube-api-access-npg4p\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.170329 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.170283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/329862e1-1940-4da0-9cca-483aec6e3486-secret-metrics-server-tls\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.170329 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.170315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/329862e1-1940-4da0-9cca-483aec6e3486-metrics-server-audit-profiles\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.170430 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.170366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/329862e1-1940-4da0-9cca-483aec6e3486-secret-metrics-server-client-certs\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.170430 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.170392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/329862e1-1940-4da0-9cca-483aec6e3486-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.170430 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.170415    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/329862e1-1940-4da0-9cca-483aec6e3486-audit-log\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.270961 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.270932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/329862e1-1940-4da0-9cca-483aec6e3486-client-ca-bundle\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.271131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.270983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npg4p\" (UniqueName: \"kubernetes.io/projected/329862e1-1940-4da0-9cca-483aec6e3486-kube-api-access-npg4p\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.271131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.271056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/329862e1-1940-4da0-9cca-483aec6e3486-secret-metrics-server-tls\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.271131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.271104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/329862e1-1940-4da0-9cca-483aec6e3486-metrics-server-audit-profiles\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.271294 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.271143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/329862e1-1940-4da0-9cca-483aec6e3486-secret-metrics-server-client-certs\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.271294 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.271193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/329862e1-1940-4da0-9cca-483aec6e3486-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.271294 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.271221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/329862e1-1940-4da0-9cca-483aec6e3486-audit-log\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.271693 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.271624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/329862e1-1940-4da0-9cca-483aec6e3486-audit-log\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.272133 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.272084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/329862e1-1940-4da0-9cca-483aec6e3486-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.272611 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.272512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/329862e1-1940-4da0-9cca-483aec6e3486-metrics-server-audit-profiles\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.273981 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.273954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/329862e1-1940-4da0-9cca-483aec6e3486-secret-metrics-server-tls\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.274094 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.274059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/329862e1-1940-4da0-9cca-483aec6e3486-secret-metrics-server-client-certs\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.274267 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.274244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/329862e1-1940-4da0-9cca-483aec6e3486-client-ca-bundle\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.287713 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.287680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npg4p\" (UniqueName: \"kubernetes.io/projected/329862e1-1940-4da0-9cca-483aec6e3486-kube-api-access-npg4p\") pod \"metrics-server-6467685b8-xgnwr\" (UID: \"329862e1-1940-4da0-9cca-483aec6e3486\") " pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.335238 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.335172    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:15.440241 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.440208    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:15.455374 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.455349    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:15.537090 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.537061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:15.620448 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.620386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h" event={"ID":"153a9d01-c1ff-471d-a43e-9387dc4917e5","Type":"ContainerStarted","Data":"12567e082f11ba0cde75304af2c3e9745c2f478b9ffda9a2f5beb837df1f1449"}
Sep 18 21:08:15.620448 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.620441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h" event={"ID":"153a9d01-c1ff-471d-a43e-9387dc4917e5","Type":"ContainerStarted","Data":"6c2656f15849e4dd8f8307e7403d55a918c988e871516c3485d3038914dbbc0e"}
Sep 18 21:08:15.620592 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.620471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h" event={"ID":"153a9d01-c1ff-471d-a43e-9387dc4917e5","Type":"ContainerStarted","Data":"59e91d9ca31ce0aa58ffabfe0958b576a54b1e6b58137c1a3061a0ff7f39c30f"}
Sep 18 21:08:15.622304 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.622278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bt5w7" event={"ID":"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa","Type":"ContainerStarted","Data":"6e0fb9a6a537c5dff662faab3a181153d3af094e27f6249243097f10ef45503f"}
Sep 18 21:08:15.622395 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.622332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bt5w7" event={"ID":"0ddfe7f6-f27f-4791-a7da-31a0c34fa3fa","Type":"ContainerStarted","Data":"404a2f87c335fe584db38976ab064e8fb297f45e324bbe471df20648e8fff908"}
Sep 18 21:08:15.624490 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.624466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-scjlc" event={"ID":"d27c900a-494e-4028-b6bc-ed4794f33e63","Type":"ContainerStarted","Data":"bf919c50a9034aea1bc6fa18be0a81cb09b03b55f20b26f9cfec60732d17584f"}
Sep 18 21:08:15.624577 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.624516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-scjlc" event={"ID":"d27c900a-494e-4028-b6bc-ed4794f33e63","Type":"ContainerStarted","Data":"0f490be8e13be18dbfb7fc7794739c99143288ba5873e913a4c2529d45c0e42d"}
Sep 18 21:08:15.626147 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.626125    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ae6fa83-229b-4b0c-b6fd-4d39f54041fd" containerID="68bf6d3d9dc8fa8bcafbcc7da50fafb09c933dd5b53fed498ccdcac0f96ce3ec" exitCode=0
Sep 18 21:08:15.626240 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.626171    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd","Type":"ContainerDied","Data":"68bf6d3d9dc8fa8bcafbcc7da50fafb09c933dd5b53fed498ccdcac0f96ce3ec"}
Sep 18 21:08:15.648636 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.648590    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-lh95h" podStartSLOduration=2.153882468 podStartE2EDuration="5.648577045s" totalImagesPullingTime="3.494694577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:08:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:15.647867043 +0000 UTC m=+133.254926600" watchObservedRunningTime="2026-09-18 21:08:15.648577045 +0000 UTC m=+133.255636602"
Sep 18 21:08:15.667801 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.667766    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bt5w7" podStartSLOduration=130.370672767 podStartE2EDuration="2m12.667754352s" totalImagesPullingTime="2.297081585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:15.666307898 +0000 UTC m=+133.273367455" watchObservedRunningTime="2026-09-18 21:08:15.667754352 +0000 UTC m=+133.274813909"
Sep 18 21:08:15.748857 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.748811    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-scjlc" podStartSLOduration=4.137043577 podStartE2EDuration="5.748795698s" totalImagesPullingTime="1.517509671s" totalInitContainerRuntime="94.24245ms" isStatefulPod=true podCreationTimestamp="2026-09-18 21:08:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:15.728451547 +0000 UTC m=+133.335511108" watchObservedRunningTime="2026-09-18 21:08:15.748795698 +0000 UTC m=+133.355855260"
Sep 18 21:08:15.750657 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.750619    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6467685b8-xgnwr"]
Sep 18 21:08:15.753172 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:08:15.753149    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329862e1_1940_4da0_9cca_483aec6e3486.slice/crio-fe32468a7c40398e3278cf3bdc5bdaa9252b37cf5df0480c0c521d8bc3945996 WatchSource:0}: Error finding container fe32468a7c40398e3278cf3bdc5bdaa9252b37cf5df0480c0c521d8bc3945996: Status 404 returned error can't find the container with id fe32468a7c40398e3278cf3bdc5bdaa9252b37cf5df0480c0c521d8bc3945996
Sep 18 21:08:15.832510 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.832476    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-58d88548b5-f4m8f"]
Sep 18 21:08:15.836499 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.836470    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:15.873110 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.873057    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58d88548b5-f4m8f"]
Sep 18 21:08:15.873189 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.873145    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:15.875890 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.875869    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 21:08:15.876957 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.876941    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 21:08:15.877216 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.877198    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 21:08:15.877289 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.877199    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 21:08:15.877289 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.877248    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 21:08:15.877397 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.877296    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-b4fdv\""
Sep 18 21:08:15.921194 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.921171    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 21:08:15.974580 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.974556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:15.977612 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.977594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:15.977696 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.977623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-telemeter-client-tls\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:15.977696 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.977659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-metrics-client-ca\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:15.977776 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.977744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-secret-telemeter-client\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:15.977813 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.977785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:15.977813 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.977808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-federate-client-tls\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:15.977892 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.977827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-serving-certs-ca-bundle\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:15.977892 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:15.977875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l775v\" (UniqueName: \"kubernetes.io/projected/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-kube-api-access-l775v\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.078629 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.078606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-metrics-client-ca\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.078771 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.078689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-secret-telemeter-client\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.078826 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.078809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.078863 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.078846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-federate-client-tls\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.078914 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.078877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-serving-certs-ca-bundle\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.078982 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.078919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l775v\" (UniqueName: \"kubernetes.io/projected/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-kube-api-access-l775v\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.078982 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.078964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.079114 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.079088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-telemeter-client-tls\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.079464 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.079408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-metrics-client-ca\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.079657 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.079612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-serving-certs-ca-bundle\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.079756 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.079712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.081531 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.081505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-telemeter-client-tls\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.081635 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.081592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-federate-client-tls\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.081721 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.081699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-secret-telemeter-client\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.081773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.081720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.089490 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.089464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l775v\" (UniqueName: \"kubernetes.io/projected/dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76-kube-api-access-l775v\") pod \"telemeter-client-58d88548b5-f4m8f\" (UID: \"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76\") " pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.182282 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.182226    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f"
Sep 18 21:08:16.428358 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.428325    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:16.489628 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.489549    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58d88548b5-f4m8f"]
Sep 18 21:08:16.490507 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:08:16.490476    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8fcf08_aaa8_4dcc_8fed_3ec01be70c76.slice/crio-5a0176c00532ba94d6006627f87214bff26a8274a1f2bc6aacbd0e9a778c7d01 WatchSource:0}: Error finding container 5a0176c00532ba94d6006627f87214bff26a8274a1f2bc6aacbd0e9a778c7d01: Status 404 returned error can't find the container with id 5a0176c00532ba94d6006627f87214bff26a8274a1f2bc6aacbd0e9a778c7d01
Sep 18 21:08:16.524115 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.524078    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:16.634369 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.634283    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f" event={"ID":"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76","Type":"ContainerStarted","Data":"5a0176c00532ba94d6006627f87214bff26a8274a1f2bc6aacbd0e9a778c7d01"}
Sep 18 21:08:16.636994 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.636933    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6467685b8-xgnwr" event={"ID":"329862e1-1940-4da0-9cca-483aec6e3486","Type":"ContainerStarted","Data":"fe32468a7c40398e3278cf3bdc5bdaa9252b37cf5df0480c0c521d8bc3945996"}
Sep 18 21:08:16.918224 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.918192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:16.923200 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.923172    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 21:08:16.940291 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.940264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.944378 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.944274    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 21:08:16.944562 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.944395    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 21:08:16.944686 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.944626    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 21:08:16.944686 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.944667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 21:08:16.944928 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.944897    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 21:08:16.944928 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.944913    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fpqob58l92n9p\""
Sep 18 21:08:16.945389 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.945361    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 21:08:16.945484 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.945444    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 21:08:16.946555 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.945614    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-s6smm\""
Sep 18 21:08:16.946555 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.945882    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 21:08:16.946555 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.945952    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 21:08:16.946555 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.946025    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 21:08:16.946555 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.946378    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 21:08:16.946555 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.946416    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 21:08:16.950026 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.949849    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 21:08:16.950800 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.950768    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 21:08:16.952952 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.952931    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 21:08:16.987658 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.987616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/784f9aa2-4755-4362-b9fb-76b1f190124d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.987820 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.987683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.987820 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.987714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.987820 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.987750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.987820 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.987781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988044 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.987844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988044 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.987896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988044 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.987921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8spw\" (UniqueName: \"kubernetes.io/projected/784f9aa2-4755-4362-b9fb-76b1f190124d-kube-api-access-l8spw\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988044 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.987984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-config\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988044 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.988022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.988092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.988119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.988148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.988177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.988205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.988241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.988271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988592 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.988300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988592 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.988358    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/784f9aa2-4755-4362-b9fb-76b1f190124d-config-out\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:16.988592 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:16.988384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-web-config\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.089521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.089484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.089521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.089529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.089844 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.089562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.089844 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.089588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.089844 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.089607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.089844 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.089631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.089844 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.089670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8spw\" (UniqueName: \"kubernetes.io/projected/784f9aa2-4755-4362-b9fb-76b1f190124d-kube-api-access-l8spw\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090107 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-config\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090171 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090232 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090232 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090315 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090315 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090315 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090421 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090421 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090421 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090421 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090555 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/784f9aa2-4755-4362-b9fb-76b1f190124d-config-out\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090555 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-web-config\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.090555 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.090520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/784f9aa2-4755-4362-b9fb-76b1f190124d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.091102 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.091057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.092306 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.091813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.092703 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.092411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.094745 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.094719    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.094957 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.094934    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.096120 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.095358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.096120 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.095842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.096289 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.096160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.097705 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.096348    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.097705 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.096436    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/784f9aa2-4755-4362-b9fb-76b1f190124d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.097705 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.096818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-config\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.097705 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.097098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.097705 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.097169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.101166 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.101147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/784f9aa2-4755-4362-b9fb-76b1f190124d-config-out\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.102256 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.102234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.102908 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.102881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/784f9aa2-4755-4362-b9fb-76b1f190124d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.103622 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.103597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.104251 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.104226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8spw\" (UniqueName: \"kubernetes.io/projected/784f9aa2-4755-4362-b9fb-76b1f190124d-kube-api-access-l8spw\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.104433 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.104386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/784f9aa2-4755-4362-b9fb-76b1f190124d-web-config\") pod \"prometheus-k8s-0\" (UID: \"784f9aa2-4755-4362-b9fb-76b1f190124d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:17.254523 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:17.254458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:18.451351 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:18.451325    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 21:08:18.453042 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:08:18.453017    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784f9aa2_4755_4362_b9fb_76b1f190124d.slice/crio-968d372effe70c4144a1107854195f8a1e7bb961b950563f9737bba24d7dc32e WatchSource:0}: Error finding container 968d372effe70c4144a1107854195f8a1e7bb961b950563f9737bba24d7dc32e: Status 404 returned error can't find the container with id 968d372effe70c4144a1107854195f8a1e7bb961b950563f9737bba24d7dc32e
Sep 18 21:08:18.648349 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:18.648246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"784f9aa2-4755-4362-b9fb-76b1f190124d","Type":"ContainerStarted","Data":"968d372effe70c4144a1107854195f8a1e7bb961b950563f9737bba24d7dc32e"}
Sep 18 21:08:18.650138 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:18.650101    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6467685b8-xgnwr" event={"ID":"329862e1-1940-4da0-9cca-483aec6e3486","Type":"ContainerStarted","Data":"b820565a837c468b3781efbf5a72e70415df24eec623519873adebd6bca9f462"}
Sep 18 21:08:18.652724 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:18.652700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd","Type":"ContainerStarted","Data":"0e665325161127a129c09c23ee70154aefbef454ce4b3847219ae2b8fecf8936"}
Sep 18 21:08:18.652848 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:18.652763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd","Type":"ContainerStarted","Data":"8e5647cfc0a94c6dff48c5e0e4922fe3d77c1d871c9ecadacd6d990c9366c373"}
Sep 18 21:08:18.681250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:18.681202    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6467685b8-xgnwr" podStartSLOduration=1.271386828 podStartE2EDuration="3.681184447s" totalImagesPullingTime="2.409797619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:18.680825054 +0000 UTC m=+136.287884614" watchObservedRunningTime="2026-09-18 21:08:18.681184447 +0000 UTC m=+136.288244004"
Sep 18 21:08:19.659290 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:19.659257    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f" event={"ID":"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76","Type":"ContainerStarted","Data":"4616958bfddb344f8a9978266dda1551a8707dfd52b9c71d8216fd1bbd5fa644"}
Sep 18 21:08:19.661000 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:19.660964    2579 generic.go:356] "Generic (PLEG): container finished" podID="784f9aa2-4755-4362-b9fb-76b1f190124d" containerID="2f310e40a3f3f2ea011405c50f0ace0f0f93dddb004076c7028151c1d6f94c3f" exitCode=0
Sep 18 21:08:19.661110 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:19.661049    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"784f9aa2-4755-4362-b9fb-76b1f190124d","Type":"ContainerDied","Data":"2f310e40a3f3f2ea011405c50f0ace0f0f93dddb004076c7028151c1d6f94c3f"}
Sep 18 21:08:19.664906 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:19.664874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd","Type":"ContainerStarted","Data":"cb19953f33f9de840ff847a5fd563c566f2acbd7a06b0cb8cd9186a11b34f493"}
Sep 18 21:08:19.665038 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:19.664953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd","Type":"ContainerStarted","Data":"6815410b9c8fd4002ccc42bbe7fbd23493254e4043f10b91fc33051ed34c1732"}
Sep 18 21:08:19.665099 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:19.665080    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd","Type":"ContainerStarted","Data":"bda3dacad028ca934fde25b3c40eb89047faa5aa84f158eaa1464f78438f42b6"}
Sep 18 21:08:20.192003 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:20.191966    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:20.293298 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:20.293243    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:20.393349 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:20.393318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:20.494185 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:20.494083    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:20.670627 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:20.670588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f" event={"ID":"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76","Type":"ContainerStarted","Data":"ea8fcf1bda01657701569fe446909f30182f564077a0041b65a8688c57e5504b"}
Sep 18 21:08:20.671031 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:20.670662    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f" event={"ID":"dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76","Type":"ContainerStarted","Data":"45b56b8f27dc668f6b31bcfebba80db0b636f353bb66b0db5772a26bc147489f"}
Sep 18 21:08:20.709432 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:20.709381    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-58d88548b5-f4m8f" podStartSLOduration=2.7983831820000002 podStartE2EDuration="5.709364634s" totalImagesPullingTime="2.910981452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:20.708048775 +0000 UTC m=+138.315108332" watchObservedRunningTime="2026-09-18 21:08:20.709364634 +0000 UTC m=+138.316424221"
Sep 18 21:08:20.770487 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:20.770458    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:20.895226 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:20.895184    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:21.364513 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:21.364480    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:21.576725 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:21.576685    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:21.679876 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:21.679788    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ae6fa83-229b-4b0c-b6fd-4d39f54041fd","Type":"ContainerStarted","Data":"1e6bb03f9e5f58343f7970402571d5ab29e2d2c95faa1f6d8dd7246eaf07e4fe"}
Sep 18 21:08:21.723557 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:21.723503    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.666523371 podStartE2EDuration="10.723489482s" totalImagesPullingTime="6.031191838s" totalInitContainerRuntime="25.774273ms" isStatefulPod=false podCreationTimestamp="2026-09-18 21:08:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:21.7209523 +0000 UTC m=+139.328011959" watchObservedRunningTime="2026-09-18 21:08:21.723489482 +0000 UTC m=+139.330549040"
Sep 18 21:08:23.096169 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:23.096136    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:24.696816 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:24.696780    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"784f9aa2-4755-4362-b9fb-76b1f190124d","Type":"ContainerStarted","Data":"1d9641e29b8257a66a38afbe127ddc95de923feb3cfcf937fb9e49b2a8d28c6b"}
Sep 18 21:08:24.697253 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:24.696849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"784f9aa2-4755-4362-b9fb-76b1f190124d","Type":"ContainerStarted","Data":"dd7c6e8bd570c686e73e2932f46c0153a7cdba1f1289fd96fe47327d6e94ef15"}
Sep 18 21:08:26.023138 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:26.023108    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:26.128501 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:26.128466    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:26.711798 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:26.711761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"784f9aa2-4755-4362-b9fb-76b1f190124d","Type":"ContainerStarted","Data":"693b8c19a459242113420a31fa1e79ea56c7290a3a32674c775ccf5c6e3cbbb6"}
Sep 18 21:08:26.711913 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:26.711894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"784f9aa2-4755-4362-b9fb-76b1f190124d","Type":"ContainerStarted","Data":"d3cf4ca891ce3c243803cbc2cb2adfe576236bc0c6ca0d26c5234ec0b9f02f26"}
Sep 18 21:08:26.711964 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:26.711950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"784f9aa2-4755-4362-b9fb-76b1f190124d","Type":"ContainerStarted","Data":"6a67bdbb812dde06fb5006a804c2c93487fa1c16edc416a64a034168f2cc7d11"}
Sep 18 21:08:26.713206 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:26.713183    2579 generic.go:356] "Generic (PLEG): container finished" podID="9f5123e8-651a-474d-a808-38dc27a1063b" containerID="acd453cc26bb1b54ea0dbd2b8dcb5c9c0a3d375234a57560f01a4a58320ca62c" exitCode=0
Sep 18 21:08:26.713290 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:26.713224    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv" event={"ID":"9f5123e8-651a-474d-a808-38dc27a1063b","Type":"ContainerDied","Data":"acd453cc26bb1b54ea0dbd2b8dcb5c9c0a3d375234a57560f01a4a58320ca62c"}
Sep 18 21:08:26.713521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:26.713505    2579 scope.go:117] "RemoveContainer" containerID="acd453cc26bb1b54ea0dbd2b8dcb5c9c0a3d375234a57560f01a4a58320ca62c"
Sep 18 21:08:27.719538 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:27.719497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xmhzv" event={"ID":"9f5123e8-651a-474d-a808-38dc27a1063b","Type":"ContainerStarted","Data":"4536d19a765cf35864ae9528a4bdcb91a8c800599a923141e7540cfc4882bc6c"}
Sep 18 21:08:27.722592 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:27.722569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"784f9aa2-4755-4362-b9fb-76b1f190124d","Type":"ContainerStarted","Data":"13b9c75269fa0979b6bcb670741b65750ed47a1208ca056eb5421924a0062eeb"}
Sep 18 21:08:27.823370 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:27.823319    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.33611109 podStartE2EDuration="11.82330239s" totalImagesPullingTime="6.440041622s" totalInitContainerRuntime="47.149678ms" isStatefulPod=false podCreationTimestamp="2026-09-18 21:08:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:08:27.822964852 +0000 UTC m=+145.430024411" watchObservedRunningTime="2026-09-18 21:08:27.82330239 +0000 UTC m=+145.430361948"
Sep 18 21:08:32.125988 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:32.125945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:32.227349 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:32.227312    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:32.254881 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:32.254847    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:08:32.327102 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:32.327070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:32.572674 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:32.572627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:32.696316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:32.696281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:32.771922 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:32.771894    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:34.814879 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:34.814838    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b" podUID="abcc2145-3a92-486e-aaa8-16b8551f6bfe" containerName="registry" containerID="cri-o://a5a61d4c4c5c0f08734c77e9df0a1ae8bd2cd59c9c392cef1e4593ac13820c91" gracePeriod=30
Sep 18 21:08:35.052701 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.052680    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:08:35.150445 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.150366    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ngmt\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-kube-api-access-8ngmt\") pod \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") "
Sep 18 21:08:35.150445 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.150428    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/abcc2145-3a92-486e-aaa8-16b8551f6bfe-trusted-ca\") pod \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") "
Sep 18 21:08:35.150673 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.150469    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls\") pod \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") "
Sep 18 21:08:35.150673 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.150487    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-certificates\") pod \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") "
Sep 18 21:08:35.150673 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.150512    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/abcc2145-3a92-486e-aaa8-16b8551f6bfe-installation-pull-secrets\") pod \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") "
Sep 18 21:08:35.150673 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.150538    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/abcc2145-3a92-486e-aaa8-16b8551f6bfe-image-registry-private-configuration\") pod \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") "
Sep 18 21:08:35.150673 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.150570    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/abcc2145-3a92-486e-aaa8-16b8551f6bfe-ca-trust-extracted\") pod \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") "
Sep 18 21:08:35.150673 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.150603    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-bound-sa-token\") pod \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\" (UID: \"abcc2145-3a92-486e-aaa8-16b8551f6bfe\") "
Sep 18 21:08:35.150985 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.150959    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abcc2145-3a92-486e-aaa8-16b8551f6bfe-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "abcc2145-3a92-486e-aaa8-16b8551f6bfe" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:08:35.151041 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.150980    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "abcc2145-3a92-486e-aaa8-16b8551f6bfe" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:08:35.152909 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.152878    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-kube-api-access-8ngmt" (OuterVolumeSpecName: "kube-api-access-8ngmt") pod "abcc2145-3a92-486e-aaa8-16b8551f6bfe" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe"). InnerVolumeSpecName "kube-api-access-8ngmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:08:35.153037 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.152998    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "abcc2145-3a92-486e-aaa8-16b8551f6bfe" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:08:35.153201 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.153179    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abcc2145-3a92-486e-aaa8-16b8551f6bfe-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "abcc2145-3a92-486e-aaa8-16b8551f6bfe" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:08:35.153263 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.153204    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "abcc2145-3a92-486e-aaa8-16b8551f6bfe" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:08:35.153329 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.153312    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abcc2145-3a92-486e-aaa8-16b8551f6bfe-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "abcc2145-3a92-486e-aaa8-16b8551f6bfe" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:08:35.159439 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.159413    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcc2145-3a92-486e-aaa8-16b8551f6bfe-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "abcc2145-3a92-486e-aaa8-16b8551f6bfe" (UID: "abcc2145-3a92-486e-aaa8-16b8551f6bfe"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:08:35.251931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.251905    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/abcc2145-3a92-486e-aaa8-16b8551f6bfe-trusted-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:35.251931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.251929    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:35.252075 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.251938    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/abcc2145-3a92-486e-aaa8-16b8551f6bfe-registry-certificates\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:35.252075 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.251948    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/abcc2145-3a92-486e-aaa8-16b8551f6bfe-installation-pull-secrets\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:35.252075 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.251958    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/abcc2145-3a92-486e-aaa8-16b8551f6bfe-image-registry-private-configuration\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:35.252075 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.251967    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/abcc2145-3a92-486e-aaa8-16b8551f6bfe-ca-trust-extracted\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:35.252075 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.251975    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-bound-sa-token\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:35.252075 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.251984    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ngmt\" (UniqueName: \"kubernetes.io/projected/abcc2145-3a92-486e-aaa8-16b8551f6bfe-kube-api-access-8ngmt\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:35.455766 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.455674    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:35.455766 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.455748    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:35.756396 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.756309    2579 generic.go:356] "Generic (PLEG): container finished" podID="abcc2145-3a92-486e-aaa8-16b8551f6bfe" containerID="a5a61d4c4c5c0f08734c77e9df0a1ae8bd2cd59c9c392cef1e4593ac13820c91" exitCode=0
Sep 18 21:08:35.756396 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.756354    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b"
Sep 18 21:08:35.756566 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.756388    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b" event={"ID":"abcc2145-3a92-486e-aaa8-16b8551f6bfe","Type":"ContainerDied","Data":"a5a61d4c4c5c0f08734c77e9df0a1ae8bd2cd59c9c392cef1e4593ac13820c91"}
Sep 18 21:08:35.756566 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.756439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84dd74ddb9-nbd9b" event={"ID":"abcc2145-3a92-486e-aaa8-16b8551f6bfe","Type":"ContainerDied","Data":"d2af6fdb04e1a01e03e38d1ef5318d8eecba523422f847de36198bbc83a5337d"}
Sep 18 21:08:35.756566 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.756462    2579 scope.go:117] "RemoveContainer" containerID="a5a61d4c4c5c0f08734c77e9df0a1ae8bd2cd59c9c392cef1e4593ac13820c91"
Sep 18 21:08:35.764920 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.764906    2579 scope.go:117] "RemoveContainer" containerID="a5a61d4c4c5c0f08734c77e9df0a1ae8bd2cd59c9c392cef1e4593ac13820c91"
Sep 18 21:08:35.765192 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:08:35.765173    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a61d4c4c5c0f08734c77e9df0a1ae8bd2cd59c9c392cef1e4593ac13820c91\": container with ID starting with a5a61d4c4c5c0f08734c77e9df0a1ae8bd2cd59c9c392cef1e4593ac13820c91 not found: ID does not exist" containerID="a5a61d4c4c5c0f08734c77e9df0a1ae8bd2cd59c9c392cef1e4593ac13820c91"
Sep 18 21:08:35.765236 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.765199    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a61d4c4c5c0f08734c77e9df0a1ae8bd2cd59c9c392cef1e4593ac13820c91"} err="failed to get container status \"a5a61d4c4c5c0f08734c77e9df0a1ae8bd2cd59c9c392cef1e4593ac13820c91\": rpc error: code = NotFound desc = could not find container \"a5a61d4c4c5c0f08734c77e9df0a1ae8bd2cd59c9c392cef1e4593ac13820c91\": container with ID starting with a5a61d4c4c5c0f08734c77e9df0a1ae8bd2cd59c9c392cef1e4593ac13820c91 not found: ID does not exist"
Sep 18 21:08:35.777138 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.777117    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84dd74ddb9-nbd9b"]
Sep 18 21:08:35.783492 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.783466    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84dd74ddb9-nbd9b"]
Sep 18 21:08:35.890252 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:35.890197    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6747499786-fwmb4" podUID="fb270b78-adc7-4082-9fd9-4b789ad7aa65" containerName="console" containerID="cri-o://f583825893a38c5d52c39cd623ccd86a18e88a88c1f7f202a756d2c09947862b" gracePeriod=15
Sep 18 21:08:36.139915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.139890    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:08:36.258714 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.258680    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-oauth-config\") pod \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") "
Sep 18 21:08:36.258714 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.258715    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-oauth-serving-cert\") pod \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") "
Sep 18 21:08:36.258915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.258749    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-serving-cert\") pod \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") "
Sep 18 21:08:36.258915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.258786    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-service-ca\") pod \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") "
Sep 18 21:08:36.258915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.258826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ztmq\" (UniqueName: \"kubernetes.io/projected/fb270b78-adc7-4082-9fd9-4b789ad7aa65-kube-api-access-2ztmq\") pod \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") "
Sep 18 21:08:36.258915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.258897    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-config\") pod \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\" (UID: \"fb270b78-adc7-4082-9fd9-4b789ad7aa65\") "
Sep 18 21:08:36.259292 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.259243    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fb270b78-adc7-4082-9fd9-4b789ad7aa65" (UID: "fb270b78-adc7-4082-9fd9-4b789ad7aa65"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:08:36.259411 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.259368    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-service-ca" (OuterVolumeSpecName: "service-ca") pod "fb270b78-adc7-4082-9fd9-4b789ad7aa65" (UID: "fb270b78-adc7-4082-9fd9-4b789ad7aa65"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:08:36.259411 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.259381    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-config" (OuterVolumeSpecName: "console-config") pod "fb270b78-adc7-4082-9fd9-4b789ad7aa65" (UID: "fb270b78-adc7-4082-9fd9-4b789ad7aa65"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:08:36.261057 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.261034    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb270b78-adc7-4082-9fd9-4b789ad7aa65-kube-api-access-2ztmq" (OuterVolumeSpecName: "kube-api-access-2ztmq") pod "fb270b78-adc7-4082-9fd9-4b789ad7aa65" (UID: "fb270b78-adc7-4082-9fd9-4b789ad7aa65"). InnerVolumeSpecName "kube-api-access-2ztmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:08:36.261278 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.261260    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fb270b78-adc7-4082-9fd9-4b789ad7aa65" (UID: "fb270b78-adc7-4082-9fd9-4b789ad7aa65"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:08:36.261338 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.261281    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fb270b78-adc7-4082-9fd9-4b789ad7aa65" (UID: "fb270b78-adc7-4082-9fd9-4b789ad7aa65"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:08:36.316904 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.316818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:36.359947 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.359924    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-config\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:36.359947 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.359946    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-oauth-config\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:36.360049 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.359958    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-oauth-serving-cert\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:36.360049 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.359966    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb270b78-adc7-4082-9fd9-4b789ad7aa65-console-serving-cert\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:36.360049 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.359974    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb270b78-adc7-4082-9fd9-4b789ad7aa65-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:36.360049 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.359983    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ztmq\" (UniqueName: \"kubernetes.io/projected/fb270b78-adc7-4082-9fd9-4b789ad7aa65-kube-api-access-2ztmq\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:08:36.395629 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.395602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:36.762761 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.762674    2579 generic.go:356] "Generic (PLEG): container finished" podID="fb270b78-adc7-4082-9fd9-4b789ad7aa65" containerID="f583825893a38c5d52c39cd623ccd86a18e88a88c1f7f202a756d2c09947862b" exitCode=0
Sep 18 21:08:36.762761 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.762722    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6747499786-fwmb4"
Sep 18 21:08:36.762966 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.762762    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6747499786-fwmb4" event={"ID":"fb270b78-adc7-4082-9fd9-4b789ad7aa65","Type":"ContainerDied","Data":"f583825893a38c5d52c39cd623ccd86a18e88a88c1f7f202a756d2c09947862b"}
Sep 18 21:08:36.762966 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.762815    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6747499786-fwmb4" event={"ID":"fb270b78-adc7-4082-9fd9-4b789ad7aa65","Type":"ContainerDied","Data":"b8a9948e26c385dee200f029ea97033a96b18f1b4bacd469bc9137498f73c5bd"}
Sep 18 21:08:36.762966 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.762840    2579 scope.go:117] "RemoveContainer" containerID="f583825893a38c5d52c39cd623ccd86a18e88a88c1f7f202a756d2c09947862b"
Sep 18 21:08:36.773808 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.773790    2579 scope.go:117] "RemoveContainer" containerID="f583825893a38c5d52c39cd623ccd86a18e88a88c1f7f202a756d2c09947862b"
Sep 18 21:08:36.774073 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:08:36.774053    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f583825893a38c5d52c39cd623ccd86a18e88a88c1f7f202a756d2c09947862b\": container with ID starting with f583825893a38c5d52c39cd623ccd86a18e88a88c1f7f202a756d2c09947862b not found: ID does not exist" containerID="f583825893a38c5d52c39cd623ccd86a18e88a88c1f7f202a756d2c09947862b"
Sep 18 21:08:36.774131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.774085    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f583825893a38c5d52c39cd623ccd86a18e88a88c1f7f202a756d2c09947862b"} err="failed to get container status \"f583825893a38c5d52c39cd623ccd86a18e88a88c1f7f202a756d2c09947862b\": rpc error: code = NotFound desc = could not find container \"f583825893a38c5d52c39cd623ccd86a18e88a88c1f7f202a756d2c09947862b\": container with ID starting with f583825893a38c5d52c39cd623ccd86a18e88a88c1f7f202a756d2c09947862b not found: ID does not exist"
Sep 18 21:08:36.797124 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.797091    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6747499786-fwmb4"]
Sep 18 21:08:36.805185 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.805162    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6747499786-fwmb4"]
Sep 18 21:08:36.920393 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.920360    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abcc2145-3a92-486e-aaa8-16b8551f6bfe" path="/var/lib/kubelet/pods/abcc2145-3a92-486e-aaa8-16b8551f6bfe/volumes"
Sep 18 21:08:36.920760 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:36.920747    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb270b78-adc7-4082-9fd9-4b789ad7aa65" path="/var/lib/kubelet/pods/fb270b78-adc7-4082-9fd9-4b789ad7aa65/volumes"
Sep 18 21:08:42.274195 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:42.274158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:42.378338 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:42.378304    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:08:55.461402 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:55.461318    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:08:55.466398 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:08:55.466372    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6467685b8-xgnwr"
Sep 18 21:09:17.255192 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:17.255155    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:09:17.274968 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:17.274936    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:09:17.946721 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:17.946692    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 21:09:26.772201 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:26.772159    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:26.835373 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:26.835343    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:38.652793 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:38.652741    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:38.713880 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:38.713839    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:38.999208 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:38.999117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:39.100689 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:39.100638    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:39.165127 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:39.165091    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:39.935933 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:39.935896    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:40.001410 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:40.001365    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:49.401846 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:49.401803    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:49.502957 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:49.502913    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:49.562382 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:49.562327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:51.344513 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:51.344477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:51.438013 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:51.437976    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:09:51.516274 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:09:51.516239    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:10:12.659602 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:10:12.659562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:10:12.766050 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:10:12.766000    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:10:38.286268 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:10:38.286177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:10:38.381299 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:10:38.381264    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:10:40.057666 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:10:40.057606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:10:42.315162 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:10:42.315089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:10:42.372552 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:10:42.372517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:10:57.087876 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:10:57.087824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:02.781791 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:02.781763    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:11:02.785878 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:02.785834    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:11:02.786678 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:02.786628    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:11:02.796399 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:02.796364    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:11:02.800080 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:02.800057    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 21:11:05.317134 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:05.317095    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:06.233343 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:06.233305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:09.556050 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:09.556012    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:09.657263 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:09.657227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:11.114031 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:11.113994    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:11.316070 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:11.316032    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:11.491711 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:11.491601    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:13.775423 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:13.775382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:13.970211 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:13.970174    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:15.928591 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:15.928549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:15.995968 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:15.995927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:20.530617 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:20.530577    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:20.627015 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:20.626978    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:20.731078 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:20.731044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:20.826327 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:20.826290    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:20.959310 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:20.959280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:21.861741 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.861706    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ksv4z"]
Sep 18 21:11:21.862119 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.862038    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb270b78-adc7-4082-9fd9-4b789ad7aa65" containerName="console"
Sep 18 21:11:21.862119 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.862049    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb270b78-adc7-4082-9fd9-4b789ad7aa65" containerName="console"
Sep 18 21:11:21.862119 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.862060    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abcc2145-3a92-486e-aaa8-16b8551f6bfe" containerName="registry"
Sep 18 21:11:21.862119 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.862066    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="abcc2145-3a92-486e-aaa8-16b8551f6bfe" containerName="registry"
Sep 18 21:11:21.862119 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.862119    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb270b78-adc7-4082-9fd9-4b789ad7aa65" containerName="console"
Sep 18 21:11:21.862287 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.862134    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="abcc2145-3a92-486e-aaa8-16b8551f6bfe" containerName="registry"
Sep 18 21:11:21.864307 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.864290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ksv4z"
Sep 18 21:11:21.866874 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.866848    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 21:11:21.866874 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.866863    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 21:11:21.868061 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.868041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7g75r\""
Sep 18 21:11:21.876695 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.876669    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ksv4z"]
Sep 18 21:11:21.933887 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.933845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15a9ca2c-ab3c-476f-95fe-6b9f61dbd357-bound-sa-token\") pod \"cert-manager-55b4d954d5-ksv4z\" (UID: \"15a9ca2c-ab3c-476f-95fe-6b9f61dbd357\") " pod="cert-manager/cert-manager-55b4d954d5-ksv4z"
Sep 18 21:11:21.934056 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:21.933945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzhwb\" (UniqueName: \"kubernetes.io/projected/15a9ca2c-ab3c-476f-95fe-6b9f61dbd357-kube-api-access-gzhwb\") pod \"cert-manager-55b4d954d5-ksv4z\" (UID: \"15a9ca2c-ab3c-476f-95fe-6b9f61dbd357\") " pod="cert-manager/cert-manager-55b4d954d5-ksv4z"
Sep 18 21:11:22.034364 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:22.034326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzhwb\" (UniqueName: \"kubernetes.io/projected/15a9ca2c-ab3c-476f-95fe-6b9f61dbd357-kube-api-access-gzhwb\") pod \"cert-manager-55b4d954d5-ksv4z\" (UID: \"15a9ca2c-ab3c-476f-95fe-6b9f61dbd357\") " pod="cert-manager/cert-manager-55b4d954d5-ksv4z"
Sep 18 21:11:22.034566 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:22.034378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15a9ca2c-ab3c-476f-95fe-6b9f61dbd357-bound-sa-token\") pod \"cert-manager-55b4d954d5-ksv4z\" (UID: \"15a9ca2c-ab3c-476f-95fe-6b9f61dbd357\") " pod="cert-manager/cert-manager-55b4d954d5-ksv4z"
Sep 18 21:11:22.044467 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:22.044439    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15a9ca2c-ab3c-476f-95fe-6b9f61dbd357-bound-sa-token\") pod \"cert-manager-55b4d954d5-ksv4z\" (UID: \"15a9ca2c-ab3c-476f-95fe-6b9f61dbd357\") " pod="cert-manager/cert-manager-55b4d954d5-ksv4z"
Sep 18 21:11:22.044773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:22.044744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzhwb\" (UniqueName: \"kubernetes.io/projected/15a9ca2c-ab3c-476f-95fe-6b9f61dbd357-kube-api-access-gzhwb\") pod \"cert-manager-55b4d954d5-ksv4z\" (UID: \"15a9ca2c-ab3c-476f-95fe-6b9f61dbd357\") " pod="cert-manager/cert-manager-55b4d954d5-ksv4z"
Sep 18 21:11:22.186421 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:22.186317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ksv4z"
Sep 18 21:11:22.456976 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:22.456949    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ksv4z"]
Sep 18 21:11:22.459455 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:11:22.459428    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15a9ca2c_ab3c_476f_95fe_6b9f61dbd357.slice/crio-38e81fe8ffb7a62a2d46193836daf3018b8ef5bb85c1a3235c4121fd30c71f75 WatchSource:0}: Error finding container 38e81fe8ffb7a62a2d46193836daf3018b8ef5bb85c1a3235c4121fd30c71f75: Status 404 returned error can't find the container with id 38e81fe8ffb7a62a2d46193836daf3018b8ef5bb85c1a3235c4121fd30c71f75
Sep 18 21:11:22.461819 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:22.461795    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:11:23.437850 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:23.437789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ksv4z" event={"ID":"15a9ca2c-ab3c-476f-95fe-6b9f61dbd357","Type":"ContainerStarted","Data":"38e81fe8ffb7a62a2d46193836daf3018b8ef5bb85c1a3235c4121fd30c71f75"}
Sep 18 21:11:23.971567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:23.971521    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:24.070795 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:24.070756    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:25.514245 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:25.514202    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n"]
Sep 18 21:11:25.516873 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:25.516847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n"
Sep 18 21:11:25.519960 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:25.519937    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 21:11:25.520934 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:25.520913    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:11:25.520934 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:25.520924    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6hlbg\""
Sep 18 21:11:25.532230 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:25.532204    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n"]
Sep 18 21:11:25.671318 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:25.671230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dcc5n\" (UID: \"bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n"
Sep 18 21:11:25.671318 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:25.671272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c2wd\" (UniqueName: \"kubernetes.io/projected/bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6-kube-api-access-4c2wd\") pod \"openshift-lws-operator-fd8ccff4c-dcc5n\" (UID: \"bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n"
Sep 18 21:11:25.772185 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:25.772143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dcc5n\" (UID: \"bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n"
Sep 18 21:11:25.772185 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:25.772184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c2wd\" (UniqueName: \"kubernetes.io/projected/bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6-kube-api-access-4c2wd\") pod \"openshift-lws-operator-fd8ccff4c-dcc5n\" (UID: \"bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n"
Sep 18 21:11:25.772559 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:25.772537    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dcc5n\" (UID: \"bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n"
Sep 18 21:11:25.783056 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:25.783028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c2wd\" (UniqueName: \"kubernetes.io/projected/bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6-kube-api-access-4c2wd\") pod \"openshift-lws-operator-fd8ccff4c-dcc5n\" (UID: \"bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n"
Sep 18 21:11:25.826027 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:25.825991    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n"
Sep 18 21:11:26.101605 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:26.101425    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n"]
Sep 18 21:11:26.104388 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:11:26.104352    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb8ea3e7_a08a_4bb5_804f_1f1bfb2838d6.slice/crio-26ee463eefb9c70d7837c2367a678786483c9d0146258dc4e4e6b9741f0d9163 WatchSource:0}: Error finding container 26ee463eefb9c70d7837c2367a678786483c9d0146258dc4e4e6b9741f0d9163: Status 404 returned error can't find the container with id 26ee463eefb9c70d7837c2367a678786483c9d0146258dc4e4e6b9741f0d9163
Sep 18 21:11:26.455669 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:26.455559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ksv4z" event={"ID":"15a9ca2c-ab3c-476f-95fe-6b9f61dbd357","Type":"ContainerStarted","Data":"22e5e56f2960d07406a4de3ef4dd81e96256a896a6a994a1e1902792bc41deb8"}
Sep 18 21:11:26.457092 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:26.457066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n" event={"ID":"bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6","Type":"ContainerStarted","Data":"26ee463eefb9c70d7837c2367a678786483c9d0146258dc4e4e6b9741f0d9163"}
Sep 18 21:11:26.481441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:26.481379    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ksv4z" podStartSLOduration=2.525294671 podStartE2EDuration="5.481358576s" totalImagesPullingTime="2.956063905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:11:26.480018547 +0000 UTC m=+324.087078107" watchObservedRunningTime="2026-09-18 21:11:26.481358576 +0000 UTC m=+324.088418134"
Sep 18 21:11:28.468109 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:28.468067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n" event={"ID":"bb8ea3e7-a08a-4bb5-804f-1f1bfb2838d6","Type":"ContainerStarted","Data":"62f14da586855f79419b0e1c3c7ea22d6658996914455b0bb0e892ffaea42bce"}
Sep 18 21:11:28.487703 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:28.487586    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dcc5n" podStartSLOduration=1.372370532 podStartE2EDuration="3.487570498s" totalImagesPullingTime="2.115199966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:11:28.485734716 +0000 UTC m=+326.092794269" watchObservedRunningTime="2026-09-18 21:11:28.487570498 +0000 UTC m=+326.094630050"
Sep 18 21:11:29.169493 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:29.169443    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:29.298582 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:29.298548    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:31.275027 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:31.274988    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:45.548974 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.548935    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:45.603791 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.603762    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"]
Sep 18 21:11:45.606374 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.606353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:45.610540 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.610479    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 21:11:45.610540 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.610509    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 21:11:45.610835 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.610565    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-n42m6\""
Sep 18 21:11:45.610835 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.610638    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 21:11:45.619338 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.619313    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"]
Sep 18 21:11:45.641808 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.641784    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:45.647910 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.647885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/267a5cd8-6809-4fc1-b8d5-f9d86cdd45de-cert\") pod \"lws-controller-manager-5d975f5588-4hxmc\" (UID: \"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de\") " pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:45.648016 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.647958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2wl\" (UniqueName: \"kubernetes.io/projected/267a5cd8-6809-4fc1-b8d5-f9d86cdd45de-kube-api-access-pf2wl\") pod \"lws-controller-manager-5d975f5588-4hxmc\" (UID: \"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de\") " pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:45.648016 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.647985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/267a5cd8-6809-4fc1-b8d5-f9d86cdd45de-manager-config\") pod \"lws-controller-manager-5d975f5588-4hxmc\" (UID: \"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de\") " pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:45.648098 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.648032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/267a5cd8-6809-4fc1-b8d5-f9d86cdd45de-metrics-cert\") pod \"lws-controller-manager-5d975f5588-4hxmc\" (UID: \"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de\") " pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:45.748991 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.748966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2wl\" (UniqueName: \"kubernetes.io/projected/267a5cd8-6809-4fc1-b8d5-f9d86cdd45de-kube-api-access-pf2wl\") pod \"lws-controller-manager-5d975f5588-4hxmc\" (UID: \"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de\") " pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:45.749132 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.748998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/267a5cd8-6809-4fc1-b8d5-f9d86cdd45de-manager-config\") pod \"lws-controller-manager-5d975f5588-4hxmc\" (UID: \"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de\") " pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:45.749132 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.749020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/267a5cd8-6809-4fc1-b8d5-f9d86cdd45de-metrics-cert\") pod \"lws-controller-manager-5d975f5588-4hxmc\" (UID: \"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de\") " pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:45.749132 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.749098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/267a5cd8-6809-4fc1-b8d5-f9d86cdd45de-cert\") pod \"lws-controller-manager-5d975f5588-4hxmc\" (UID: \"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de\") " pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:45.749773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.749749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/267a5cd8-6809-4fc1-b8d5-f9d86cdd45de-manager-config\") pod \"lws-controller-manager-5d975f5588-4hxmc\" (UID: \"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de\") " pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:45.751742 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.751716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/267a5cd8-6809-4fc1-b8d5-f9d86cdd45de-metrics-cert\") pod \"lws-controller-manager-5d975f5588-4hxmc\" (UID: \"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de\") " pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:45.751837 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.751716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/267a5cd8-6809-4fc1-b8d5-f9d86cdd45de-cert\") pod \"lws-controller-manager-5d975f5588-4hxmc\" (UID: \"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de\") " pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:45.758596 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.758574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2wl\" (UniqueName: \"kubernetes.io/projected/267a5cd8-6809-4fc1-b8d5-f9d86cdd45de-kube-api-access-pf2wl\") pod \"lws-controller-manager-5d975f5588-4hxmc\" (UID: \"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de\") " pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:45.915729 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:45.915625    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:46.186487 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:46.186414    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"]
Sep 18 21:11:46.189517 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:11:46.189486    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267a5cd8_6809_4fc1_b8d5_f9d86cdd45de.slice/crio-2e28510e314871a32a9ec28d2768c590982922b87beb5a117114f6e8ca6d16f2 WatchSource:0}: Error finding container 2e28510e314871a32a9ec28d2768c590982922b87beb5a117114f6e8ca6d16f2: Status 404 returned error can't find the container with id 2e28510e314871a32a9ec28d2768c590982922b87beb5a117114f6e8ca6d16f2
Sep 18 21:11:46.553673 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:46.553619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc" event={"ID":"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de","Type":"ContainerStarted","Data":"2e28510e314871a32a9ec28d2768c590982922b87beb5a117114f6e8ca6d16f2"}
Sep 18 21:11:47.172428 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:47.172382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:47.312254 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:47.312213    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:47.744264 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:47.744224    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:48.010927 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.010831    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:48.209515 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.209482    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"]
Sep 18 21:11:48.212365 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.212341    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.216743 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.216715    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 21:11:48.216902 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.216712    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 21:11:48.216902 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.216833    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 21:11:48.216902 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.216864    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 21:11:48.219683 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.219664    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6hf49\""
Sep 18 21:11:48.235446 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.235414    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"]
Sep 18 21:11:48.277775 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.277680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dg4v\" (UniqueName: \"kubernetes.io/projected/cf062a5c-8ceb-4610-972e-63c5af6b96aa-kube-api-access-2dg4v\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.277931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.277801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf062a5c-8ceb-4610-972e-63c5af6b96aa-tmp\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.277931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.277854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf062a5c-8ceb-4610-972e-63c5af6b96aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.277931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.277892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf062a5c-8ceb-4610-972e-63c5af6b96aa-manifests\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.277931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.277912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf062a5c-8ceb-4610-972e-63c5af6b96aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.379131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.379094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf062a5c-8ceb-4610-972e-63c5af6b96aa-tmp\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.379131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.379143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf062a5c-8ceb-4610-972e-63c5af6b96aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.379378 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.379181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf062a5c-8ceb-4610-972e-63c5af6b96aa-manifests\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.379378 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.379209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf062a5c-8ceb-4610-972e-63c5af6b96aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.379378 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.379282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dg4v\" (UniqueName: \"kubernetes.io/projected/cf062a5c-8ceb-4610-972e-63c5af6b96aa-kube-api-access-2dg4v\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.379654 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.379611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf062a5c-8ceb-4610-972e-63c5af6b96aa-manifests\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.381703 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.381681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf062a5c-8ceb-4610-972e-63c5af6b96aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.381932 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.381912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf062a5c-8ceb-4610-972e-63c5af6b96aa-tmp\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.382111 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.382092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf062a5c-8ceb-4610-972e-63c5af6b96aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.398429 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.398399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dg4v\" (UniqueName: \"kubernetes.io/projected/cf062a5c-8ceb-4610-972e-63c5af6b96aa-kube-api-access-2dg4v\") pod \"opendatahub-operator-controller-manager-75f94bc69c-4rb8s\" (UID: \"cf062a5c-8ceb-4610-972e-63c5af6b96aa\") " pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.523702 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.523640    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:48.566359 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.566303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc" event={"ID":"267a5cd8-6809-4fc1-b8d5-f9d86cdd45de","Type":"ContainerStarted","Data":"9fb85f15af3f9c2212f7a0ccc348c9792733b16dc85d24acaf2d0feeb1de2312"}
Sep 18 21:11:48.566562 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.566411    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:11:48.589535 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.589486    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc" podStartSLOduration=1.888592979 podStartE2EDuration="3.589469341s" totalImagesPullingTime="1.700876362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:11:48.588423015 +0000 UTC m=+346.195482574" watchObservedRunningTime="2026-09-18 21:11:48.589469341 +0000 UTC m=+346.196528899"
Sep 18 21:11:48.803967 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:48.803930    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"]
Sep 18 21:11:48.807739 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:11:48.807707    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf062a5c_8ceb_4610_972e_63c5af6b96aa.slice/crio-ecfcbfc468d294d37c64156a0ebe04d28c192e141b4f62aa8f5af0cf1ed9f09b WatchSource:0}: Error finding container ecfcbfc468d294d37c64156a0ebe04d28c192e141b4f62aa8f5af0cf1ed9f09b: Status 404 returned error can't find the container with id ecfcbfc468d294d37c64156a0ebe04d28c192e141b4f62aa8f5af0cf1ed9f09b
Sep 18 21:11:49.245746 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:49.245661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:11:49.574167 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:49.574125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s" event={"ID":"cf062a5c-8ceb-4610-972e-63c5af6b96aa","Type":"ContainerStarted","Data":"ecfcbfc468d294d37c64156a0ebe04d28c192e141b4f62aa8f5af0cf1ed9f09b"}
Sep 18 21:11:52.590953 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:52.590914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s" event={"ID":"cf062a5c-8ceb-4610-972e-63c5af6b96aa","Type":"ContainerStarted","Data":"095e27e4c43d3826cb343b11c2b8d2b668d559ec792024ab88e72c0aa925313c"}
Sep 18 21:11:53.596724 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:53.596688    2579 generic.go:356] "Generic (PLEG): container finished" podID="cf062a5c-8ceb-4610-972e-63c5af6b96aa" containerID="095e27e4c43d3826cb343b11c2b8d2b668d559ec792024ab88e72c0aa925313c" exitCode=0
Sep 18 21:11:53.597099 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:53.596779    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s" event={"ID":"cf062a5c-8ceb-4610-972e-63c5af6b96aa","Type":"ContainerDied","Data":"095e27e4c43d3826cb343b11c2b8d2b668d559ec792024ab88e72c0aa925313c"}
Sep 18 21:11:54.602851 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:54.602813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s" event={"ID":"cf062a5c-8ceb-4610-972e-63c5af6b96aa","Type":"ContainerStarted","Data":"48a026f101b091302062dc328483c32358ad55a636542cf51923fde3b2ef3e2f"}
Sep 18 21:11:54.603277 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:54.602909    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:11:54.628179 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:54.628118    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s" podStartSLOduration=2.170341296 podStartE2EDuration="6.628095904s" totalImagesPullingTime="3.393303974s" totalInitContainerRuntime="1.064450634s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:11:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:11:54.624551664 +0000 UTC m=+352.231611214" watchObservedRunningTime="2026-09-18 21:11:54.628095904 +0000 UTC m=+352.235155463"
Sep 18 21:11:59.577166 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:11:59.577089    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d975f5588-4hxmc"
Sep 18 21:12:00.143612 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:00.143575    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:00.248117 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:00.248084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:05.609703 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:05.609673    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-75f94bc69c-4rb8s"
Sep 18 21:12:07.907347 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:07.907307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:08.075400 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.075362    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:08.442077 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.442035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:08.503295 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.503254    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"]
Sep 18 21:12:08.513330 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.513300    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"
Sep 18 21:12:08.520541 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.520305    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 21:12:08.521201 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.521178    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 21:12:08.521296 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.521178    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5grtt\""
Sep 18 21:12:08.521296 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.521244    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 21:12:08.521296 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.521182    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 21:12:08.526088 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.526059    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"]
Sep 18 21:12:08.657766 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.657731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b1e7d0-2639-47b4-ba8d-815c44e99e30-tls-certs\") pod \"kube-auth-proxy-85cdd48bb5-vg2xq\" (UID: \"90b1e7d0-2639-47b4-ba8d-815c44e99e30\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"
Sep 18 21:12:08.657967 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.657835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90b1e7d0-2639-47b4-ba8d-815c44e99e30-tmp\") pod \"kube-auth-proxy-85cdd48bb5-vg2xq\" (UID: \"90b1e7d0-2639-47b4-ba8d-815c44e99e30\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"
Sep 18 21:12:08.657967 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.657866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhmq\" (UniqueName: \"kubernetes.io/projected/90b1e7d0-2639-47b4-ba8d-815c44e99e30-kube-api-access-bqhmq\") pod \"kube-auth-proxy-85cdd48bb5-vg2xq\" (UID: \"90b1e7d0-2639-47b4-ba8d-815c44e99e30\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"
Sep 18 21:12:08.658298 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.658271    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:08.759116 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.759019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90b1e7d0-2639-47b4-ba8d-815c44e99e30-tmp\") pod \"kube-auth-proxy-85cdd48bb5-vg2xq\" (UID: \"90b1e7d0-2639-47b4-ba8d-815c44e99e30\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"
Sep 18 21:12:08.759116 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.759062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhmq\" (UniqueName: \"kubernetes.io/projected/90b1e7d0-2639-47b4-ba8d-815c44e99e30-kube-api-access-bqhmq\") pod \"kube-auth-proxy-85cdd48bb5-vg2xq\" (UID: \"90b1e7d0-2639-47b4-ba8d-815c44e99e30\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"
Sep 18 21:12:08.759116 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.759112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b1e7d0-2639-47b4-ba8d-815c44e99e30-tls-certs\") pod \"kube-auth-proxy-85cdd48bb5-vg2xq\" (UID: \"90b1e7d0-2639-47b4-ba8d-815c44e99e30\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"
Sep 18 21:12:08.761427 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.761400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90b1e7d0-2639-47b4-ba8d-815c44e99e30-tmp\") pod \"kube-auth-proxy-85cdd48bb5-vg2xq\" (UID: \"90b1e7d0-2639-47b4-ba8d-815c44e99e30\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"
Sep 18 21:12:08.761557 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.761538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90b1e7d0-2639-47b4-ba8d-815c44e99e30-tls-certs\") pod \"kube-auth-proxy-85cdd48bb5-vg2xq\" (UID: \"90b1e7d0-2639-47b4-ba8d-815c44e99e30\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"
Sep 18 21:12:08.774510 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.774478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhmq\" (UniqueName: \"kubernetes.io/projected/90b1e7d0-2639-47b4-ba8d-815c44e99e30-kube-api-access-bqhmq\") pod \"kube-auth-proxy-85cdd48bb5-vg2xq\" (UID: \"90b1e7d0-2639-47b4-ba8d-815c44e99e30\") " pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"
Sep 18 21:12:08.823515 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:08.823478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"
Sep 18 21:12:09.114254 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:09.114227    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq"]
Sep 18 21:12:09.116195 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:12:09.116160    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b1e7d0_2639_47b4_ba8d_815c44e99e30.slice/crio-a1f7dda3885c60a079f5d19f34fb15807cb5f1e9e592ef9a388ad040e4b1adeb WatchSource:0}: Error finding container a1f7dda3885c60a079f5d19f34fb15807cb5f1e9e592ef9a388ad040e4b1adeb: Status 404 returned error can't find the container with id a1f7dda3885c60a079f5d19f34fb15807cb5f1e9e592ef9a388ad040e4b1adeb
Sep 18 21:12:09.674454 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:09.674408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq" event={"ID":"90b1e7d0-2639-47b4-ba8d-815c44e99e30","Type":"ContainerStarted","Data":"a1f7dda3885c60a079f5d19f34fb15807cb5f1e9e592ef9a388ad040e4b1adeb"}
Sep 18 21:12:10.655671 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:10.655603    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:10.751232 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:10.751191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:12.106838 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:12.106803    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:12.692620 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:12.692583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq" event={"ID":"90b1e7d0-2639-47b4-ba8d-815c44e99e30","Type":"ContainerStarted","Data":"9d07ceb96db73e241d60fdb3658b0c85caf7bf1885eb48bbf433220091cf3be4"}
Sep 18 21:12:12.714616 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:12.714553    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85cdd48bb5-vg2xq" podStartSLOduration=2.092231723 podStartE2EDuration="4.714534301s" totalImagesPullingTime="2.622302578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:12:12.714220283 +0000 UTC m=+370.321279843" watchObservedRunningTime="2026-09-18 21:12:12.714534301 +0000 UTC m=+370.321593861"
Sep 18 21:12:15.223618 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:15.223584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:15.318749 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:15.318714    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:15.983673 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:15.983629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:16.080161 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:16.080126    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:16.191547 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:16.191518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:18.912497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:18.912456    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:19.011983 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:19.011955    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:19.113253 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:19.113208    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:19.511380 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:19.511352    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:19.585455 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:19.585418    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:29.249438 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:29.249379    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:29.353149 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:29.353105    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:29.422436 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:29.422395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:29.824667 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:29.824620    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:30.647144 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:30.647106    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:30.784368 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:30.784332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:55.550536 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:55.550491    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:55.652504 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:55.652440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:57.518633 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:57.518575    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:12:57.578341 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:12:57.578308    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:12.116251 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:12.116216    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:12.939498 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:12.939458    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:20.195114 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:20.195065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:20.610639 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:20.610586    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:23.810622 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:23.810589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:23.958064 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:23.958033    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:33.493953 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:33.493871    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:41.974546 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:41.974506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:44.569619 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:44.569579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:44.975277 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:44.975185    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:46.448513 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:46.448477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:47.611398 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:47.611360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:47.835660 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:47.835606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:49.300989 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:49.300952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:50.187872 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:50.187841    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:50.585850 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:50.585812    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:50.991319 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:50.991229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:51.602932 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:51.602899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:51.740900 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:51.740862    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:55.373110 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:55.373074    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:55.525124 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:55.525084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:13:59.905376 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:13:59.905341    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:00.026323 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:00.026291    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:00.141166 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:00.141125    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:01.383242 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:01.383204    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:01.715674 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:01.715563    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:02.052461 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:02.052430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:02.161229 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:02.161196    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:08.572736 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:08.572700    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:08.674973 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:08.674941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:08.768511 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:08.768477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:12.025591 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.025558    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"]
Sep 18 21:14:12.028772 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.028746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"
Sep 18 21:14:12.031921 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.031898    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 21:14:12.032030 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.032010    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 21:14:12.032089 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.032039    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zlvvm\""
Sep 18 21:14:12.042052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.042031    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"]
Sep 18 21:14:12.074723 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.074691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xx52\" (UniqueName: \"kubernetes.io/projected/0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4-kube-api-access-4xx52\") pod \"kuadrant-operator-controller-manager-5f685c496f-x8qj9\" (UID: \"0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"
Sep 18 21:14:12.074893 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.074781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-x8qj9\" (UID: \"0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"
Sep 18 21:14:12.175564 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.175514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-x8qj9\" (UID: \"0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"
Sep 18 21:14:12.175766 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.175609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xx52\" (UniqueName: \"kubernetes.io/projected/0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4-kube-api-access-4xx52\") pod \"kuadrant-operator-controller-manager-5f685c496f-x8qj9\" (UID: \"0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"
Sep 18 21:14:12.175946 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.175914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-x8qj9\" (UID: \"0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"
Sep 18 21:14:12.186746 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.186713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xx52\" (UniqueName: \"kubernetes.io/projected/0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4-kube-api-access-4xx52\") pod \"kuadrant-operator-controller-manager-5f685c496f-x8qj9\" (UID: \"0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"
Sep 18 21:14:12.339326 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.339288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"
Sep 18 21:14:12.636752 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.636724    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"]
Sep 18 21:14:12.639407 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:14:12.639379    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a668abf_6e4a_4b5c_a70a_27b3b8fc44d4.slice/crio-fa6af6c1ede227c3d7cd25c2b121781d0a86dcea5155f0025d0a45ba2999eb55 WatchSource:0}: Error finding container fa6af6c1ede227c3d7cd25c2b121781d0a86dcea5155f0025d0a45ba2999eb55: Status 404 returned error can't find the container with id fa6af6c1ede227c3d7cd25c2b121781d0a86dcea5155f0025d0a45ba2999eb55
Sep 18 21:14:12.857265 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.855988    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"]
Sep 18 21:14:12.870917 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.870884    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"]
Sep 18 21:14:12.973274 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.973199    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"]
Sep 18 21:14:12.977677 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:12.977639    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"
Sep 18 21:14:13.062397 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:13.062362    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"]
Sep 18 21:14:13.084540 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:13.084508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zs92\" (UniqueName: \"kubernetes.io/projected/c08afb7d-527b-4953-b235-28fa09ff2cbb-kube-api-access-8zs92\") pod \"kuadrant-operator-controller-manager-5f685c496f-cqhx4\" (UID: \"c08afb7d-527b-4953-b235-28fa09ff2cbb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"
Sep 18 21:14:13.084747 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:13.084555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c08afb7d-527b-4953-b235-28fa09ff2cbb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-cqhx4\" (UID: \"c08afb7d-527b-4953-b235-28fa09ff2cbb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"
Sep 18 21:14:13.186028 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:13.185983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zs92\" (UniqueName: \"kubernetes.io/projected/c08afb7d-527b-4953-b235-28fa09ff2cbb-kube-api-access-8zs92\") pod \"kuadrant-operator-controller-manager-5f685c496f-cqhx4\" (UID: \"c08afb7d-527b-4953-b235-28fa09ff2cbb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"
Sep 18 21:14:13.186217 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:13.186055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c08afb7d-527b-4953-b235-28fa09ff2cbb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-cqhx4\" (UID: \"c08afb7d-527b-4953-b235-28fa09ff2cbb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"
Sep 18 21:14:13.186469 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:13.186447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c08afb7d-527b-4953-b235-28fa09ff2cbb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-cqhx4\" (UID: \"c08afb7d-527b-4953-b235-28fa09ff2cbb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"
Sep 18 21:14:13.205386 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:13.205353    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zs92\" (UniqueName: \"kubernetes.io/projected/c08afb7d-527b-4953-b235-28fa09ff2cbb-kube-api-access-8zs92\") pod \"kuadrant-operator-controller-manager-5f685c496f-cqhx4\" (UID: \"c08afb7d-527b-4953-b235-28fa09ff2cbb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"
Sep 18 21:14:13.290783 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:13.290753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"
Sep 18 21:14:13.608831 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:13.608701    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"]
Sep 18 21:14:13.613665 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:14:13.613610    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08afb7d_527b_4953_b235_28fa09ff2cbb.slice/crio-fa4619f09aa09724dabac9f3709bebee9efb9d868d837d2050aecc5902bef17c WatchSource:0}: Error finding container fa4619f09aa09724dabac9f3709bebee9efb9d868d837d2050aecc5902bef17c: Status 404 returned error can't find the container with id fa4619f09aa09724dabac9f3709bebee9efb9d868d837d2050aecc5902bef17c
Sep 18 21:14:14.262690 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:14.262627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4" event={"ID":"c08afb7d-527b-4953-b235-28fa09ff2cbb","Type":"ContainerStarted","Data":"fa4619f09aa09724dabac9f3709bebee9efb9d868d837d2050aecc5902bef17c"}
Sep 18 21:14:15.905227 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:15.905191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:16.102977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:16.102941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:17.281142 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:17.281049    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4" event={"ID":"c08afb7d-527b-4953-b235-28fa09ff2cbb","Type":"ContainerStarted","Data":"0756ce954ff2cb9f6e595451c4aeaee73d2d86f798639aa45534a14e6daaf744"}
Sep 18 21:14:17.281545 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:17.281141    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"
Sep 18 21:14:17.282966 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:17.282920    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9" podUID="0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4" containerName="manager" containerID="cri-o://b06fe7671358d6610ff83ada82101314600aa1002847c6881bd210025de21e41" gracePeriod=2
Sep 18 21:14:17.332253 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:17.332195    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4" podStartSLOduration=2.108139884 podStartE2EDuration="5.332180243s" totalImagesPullingTime="3.224040359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:14:17.329854826 +0000 UTC m=+494.936914384" watchObservedRunningTime="2026-09-18 21:14:17.332180243 +0000 UTC m=+494.939239798"
Sep 18 21:14:17.531015 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:17.530989    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"
Sep 18 21:14:17.535237 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:14:17.535211    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-x8qj9\" is forbidden: User \"system:node:ip-10-0-130-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-92.ec2.internal' and this object" podUID="0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"
Sep 18 21:14:17.631432 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:17.631403    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4-extensions-socket-volume\") pod \"0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4\" (UID: \"0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4\") "
Sep 18 21:14:17.631588 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:17.631522    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xx52\" (UniqueName: \"kubernetes.io/projected/0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4-kube-api-access-4xx52\") pod \"0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4\" (UID: \"0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4\") "
Sep 18 21:14:17.631744 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:17.631721    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4" (UID: "0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:14:17.631821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:17.631794    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4-extensions-socket-volume\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:14:17.633682 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:17.633639    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4-kube-api-access-4xx52" (OuterVolumeSpecName: "kube-api-access-4xx52") pod "0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4" (UID: "0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4"). InnerVolumeSpecName "kube-api-access-4xx52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:14:17.732951 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:17.732915    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xx52\" (UniqueName: \"kubernetes.io/projected/0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4-kube-api-access-4xx52\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:14:18.288433 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:18.288400    2579 generic.go:356] "Generic (PLEG): container finished" podID="0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4" containerID="b06fe7671358d6610ff83ada82101314600aa1002847c6881bd210025de21e41" exitCode=2
Sep 18 21:14:18.288824 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:18.288440    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"
Sep 18 21:14:18.288824 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:18.288482    2579 scope.go:117] "RemoveContainer" containerID="b06fe7671358d6610ff83ada82101314600aa1002847c6881bd210025de21e41"
Sep 18 21:14:18.291265 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:14:18.291242    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-x8qj9\" is forbidden: User \"system:node:ip-10-0-130-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-92.ec2.internal' and this object" podUID="0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"
Sep 18 21:14:18.297596 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:18.297579    2579 scope.go:117] "RemoveContainer" containerID="b06fe7671358d6610ff83ada82101314600aa1002847c6881bd210025de21e41"
Sep 18 21:14:18.297888 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:14:18.297867    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06fe7671358d6610ff83ada82101314600aa1002847c6881bd210025de21e41\": container with ID starting with b06fe7671358d6610ff83ada82101314600aa1002847c6881bd210025de21e41 not found: ID does not exist" containerID="b06fe7671358d6610ff83ada82101314600aa1002847c6881bd210025de21e41"
Sep 18 21:14:18.297944 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:18.297897    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06fe7671358d6610ff83ada82101314600aa1002847c6881bd210025de21e41"} err="failed to get container status \"b06fe7671358d6610ff83ada82101314600aa1002847c6881bd210025de21e41\": rpc error: code = NotFound desc = could not find container \"b06fe7671358d6610ff83ada82101314600aa1002847c6881bd210025de21e41\": container with ID starting with b06fe7671358d6610ff83ada82101314600aa1002847c6881bd210025de21e41 not found: ID does not exist"
Sep 18 21:14:18.301712 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:14:18.301690    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-x8qj9\" is forbidden: User \"system:node:ip-10-0-130-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-92.ec2.internal' and this object" podUID="0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x8qj9"
Sep 18 21:14:18.921730 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:18.921687    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4" path="/var/lib/kubelet/pods/0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4/volumes"
Sep 18 21:14:19.100856 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:19.100819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:19.209096 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:19.209009    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:19.310492 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:19.310460    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:22.832476 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:22.832445    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:22.932832 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:22.932806    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:28.290232 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:28.290199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"
Sep 18 21:14:28.903546 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:28.903461    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:28.996118 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:28.996087    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:29.106450 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:29.106404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:38.994633 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:38.994589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:40.820953 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:40.820917    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"]
Sep 18 21:14:40.821368 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:40.821141    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4" podUID="c08afb7d-527b-4953-b235-28fa09ff2cbb" containerName="manager" containerID="cri-o://0756ce954ff2cb9f6e595451c4aeaee73d2d86f798639aa45534a14e6daaf744" gracePeriod=10
Sep 18 21:14:41.073839 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.073776    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"
Sep 18 21:14:41.244961 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.244921    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zs92\" (UniqueName: \"kubernetes.io/projected/c08afb7d-527b-4953-b235-28fa09ff2cbb-kube-api-access-8zs92\") pod \"c08afb7d-527b-4953-b235-28fa09ff2cbb\" (UID: \"c08afb7d-527b-4953-b235-28fa09ff2cbb\") "
Sep 18 21:14:41.244961 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.244969    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c08afb7d-527b-4953-b235-28fa09ff2cbb-extensions-socket-volume\") pod \"c08afb7d-527b-4953-b235-28fa09ff2cbb\" (UID: \"c08afb7d-527b-4953-b235-28fa09ff2cbb\") "
Sep 18 21:14:41.245498 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.245473    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08afb7d-527b-4953-b235-28fa09ff2cbb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c08afb7d-527b-4953-b235-28fa09ff2cbb" (UID: "c08afb7d-527b-4953-b235-28fa09ff2cbb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:14:41.247233 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.247205    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08afb7d-527b-4953-b235-28fa09ff2cbb-kube-api-access-8zs92" (OuterVolumeSpecName: "kube-api-access-8zs92") pod "c08afb7d-527b-4953-b235-28fa09ff2cbb" (UID: "c08afb7d-527b-4953-b235-28fa09ff2cbb"). InnerVolumeSpecName "kube-api-access-8zs92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:14:41.346423 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.346339    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zs92\" (UniqueName: \"kubernetes.io/projected/c08afb7d-527b-4953-b235-28fa09ff2cbb-kube-api-access-8zs92\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:14:41.346423 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.346367    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c08afb7d-527b-4953-b235-28fa09ff2cbb-extensions-socket-volume\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:14:41.401972 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.401932    2579 generic.go:356] "Generic (PLEG): container finished" podID="c08afb7d-527b-4953-b235-28fa09ff2cbb" containerID="0756ce954ff2cb9f6e595451c4aeaee73d2d86f798639aa45534a14e6daaf744" exitCode=0
Sep 18 21:14:41.402161 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.401984    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"
Sep 18 21:14:41.402161 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.402028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4" event={"ID":"c08afb7d-527b-4953-b235-28fa09ff2cbb","Type":"ContainerDied","Data":"0756ce954ff2cb9f6e595451c4aeaee73d2d86f798639aa45534a14e6daaf744"}
Sep 18 21:14:41.402161 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.402091    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4" event={"ID":"c08afb7d-527b-4953-b235-28fa09ff2cbb","Type":"ContainerDied","Data":"fa4619f09aa09724dabac9f3709bebee9efb9d868d837d2050aecc5902bef17c"}
Sep 18 21:14:41.402161 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.402125    2579 scope.go:117] "RemoveContainer" containerID="0756ce954ff2cb9f6e595451c4aeaee73d2d86f798639aa45534a14e6daaf744"
Sep 18 21:14:41.410735 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.410713    2579 scope.go:117] "RemoveContainer" containerID="0756ce954ff2cb9f6e595451c4aeaee73d2d86f798639aa45534a14e6daaf744"
Sep 18 21:14:41.411049 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:14:41.411031    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0756ce954ff2cb9f6e595451c4aeaee73d2d86f798639aa45534a14e6daaf744\": container with ID starting with 0756ce954ff2cb9f6e595451c4aeaee73d2d86f798639aa45534a14e6daaf744 not found: ID does not exist" containerID="0756ce954ff2cb9f6e595451c4aeaee73d2d86f798639aa45534a14e6daaf744"
Sep 18 21:14:41.411109 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.411059    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0756ce954ff2cb9f6e595451c4aeaee73d2d86f798639aa45534a14e6daaf744"} err="failed to get container status \"0756ce954ff2cb9f6e595451c4aeaee73d2d86f798639aa45534a14e6daaf744\": rpc error: code = NotFound desc = could not find container \"0756ce954ff2cb9f6e595451c4aeaee73d2d86f798639aa45534a14e6daaf744\": container with ID starting with 0756ce954ff2cb9f6e595451c4aeaee73d2d86f798639aa45534a14e6daaf744 not found: ID does not exist"
Sep 18 21:14:41.443350 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.443314    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"]
Sep 18 21:14:41.466222 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:41.466188    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-cqhx4"]
Sep 18 21:14:42.921340 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:42.921305    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c08afb7d-527b-4953-b235-28fa09ff2cbb" path="/var/lib/kubelet/pods/c08afb7d-527b-4953-b235-28fa09ff2cbb/volumes"
Sep 18 21:14:44.932546 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:44.932494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:45.286420 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:45.286392    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:49.031858 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:49.031822    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:49.132359 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:49.132304    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:56.000991 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.000907    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5bnx7"]
Sep 18 21:14:56.001435 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.001265    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4" containerName="manager"
Sep 18 21:14:56.001435 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.001276    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4" containerName="manager"
Sep 18 21:14:56.001435 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.001298    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c08afb7d-527b-4953-b235-28fa09ff2cbb" containerName="manager"
Sep 18 21:14:56.001435 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.001303    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c08afb7d-527b-4953-b235-28fa09ff2cbb" containerName="manager"
Sep 18 21:14:56.001435 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.001350    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c08afb7d-527b-4953-b235-28fa09ff2cbb" containerName="manager"
Sep 18 21:14:56.001435 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.001361    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a668abf-6e4a-4b5c-a70a-27b3b8fc44d4" containerName="manager"
Sep 18 21:14:56.004155 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.004133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5bnx7"
Sep 18 21:14:56.009311 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.009286    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 21:14:56.009929 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.009909    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 21:14:56.010051 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.009935    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 21:14:56.011702 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.011684    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sj6pb\""
Sep 18 21:14:56.053342 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.053306    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:56.065585 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.065552    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5bnx7"]
Sep 18 21:14:56.067419 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.067390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a0838c7-b955-430e-b22c-257bde994f83-config-file\") pod \"limitador-limitador-8588749977-5bnx7\" (UID: \"9a0838c7-b955-430e-b22c-257bde994f83\") " pod="kuadrant-system/limitador-limitador-8588749977-5bnx7"
Sep 18 21:14:56.067537 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.067458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mt4w\" (UniqueName: \"kubernetes.io/projected/9a0838c7-b955-430e-b22c-257bde994f83-kube-api-access-5mt4w\") pod \"limitador-limitador-8588749977-5bnx7\" (UID: \"9a0838c7-b955-430e-b22c-257bde994f83\") " pod="kuadrant-system/limitador-limitador-8588749977-5bnx7"
Sep 18 21:14:56.097062 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.097032    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5bnx7"]
Sep 18 21:14:56.168374 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.168342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a0838c7-b955-430e-b22c-257bde994f83-config-file\") pod \"limitador-limitador-8588749977-5bnx7\" (UID: \"9a0838c7-b955-430e-b22c-257bde994f83\") " pod="kuadrant-system/limitador-limitador-8588749977-5bnx7"
Sep 18 21:14:56.168534 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.168393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mt4w\" (UniqueName: \"kubernetes.io/projected/9a0838c7-b955-430e-b22c-257bde994f83-kube-api-access-5mt4w\") pod \"limitador-limitador-8588749977-5bnx7\" (UID: \"9a0838c7-b955-430e-b22c-257bde994f83\") " pod="kuadrant-system/limitador-limitador-8588749977-5bnx7"
Sep 18 21:14:56.168996 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.168977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a0838c7-b955-430e-b22c-257bde994f83-config-file\") pod \"limitador-limitador-8588749977-5bnx7\" (UID: \"9a0838c7-b955-430e-b22c-257bde994f83\") " pod="kuadrant-system/limitador-limitador-8588749977-5bnx7"
Sep 18 21:14:56.179332 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.179301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:56.180836 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.180819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mt4w\" (UniqueName: \"kubernetes.io/projected/9a0838c7-b955-430e-b22c-257bde994f83-kube-api-access-5mt4w\") pod \"limitador-limitador-8588749977-5bnx7\" (UID: \"9a0838c7-b955-430e-b22c-257bde994f83\") " pod="kuadrant-system/limitador-limitador-8588749977-5bnx7"
Sep 18 21:14:56.313996 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.313964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5bnx7"
Sep 18 21:14:56.615059 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:56.614986    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5bnx7"]
Sep 18 21:14:56.618686 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:14:56.618624    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0838c7_b955_430e_b22c_257bde994f83.slice/crio-6e3ab43f1ce091da73075beb725442031d73a9150ab306942ce36b0a00ffabbe WatchSource:0}: Error finding container 6e3ab43f1ce091da73075beb725442031d73a9150ab306942ce36b0a00ffabbe: Status 404 returned error can't find the container with id 6e3ab43f1ce091da73075beb725442031d73a9150ab306942ce36b0a00ffabbe
Sep 18 21:14:57.083011 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:57.082979    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b6rjt"]
Sep 18 21:14:57.086283 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:57.086262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b6rjt"
Sep 18 21:14:57.089703 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:57.089681    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-95fsz\""
Sep 18 21:14:57.122131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:57.122103    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b6rjt"]
Sep 18 21:14:57.165895 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:57.165869    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:57.177054 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:57.177031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrzr4\" (UniqueName: \"kubernetes.io/projected/1d74b1a4-b10a-4477-8515-edb8e5506eb3-kube-api-access-xrzr4\") pod \"authorino-7bcfd8f9cd-b6rjt\" (UID: \"1d74b1a4-b10a-4477-8515-edb8e5506eb3\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b6rjt"
Sep 18 21:14:57.278470 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:57.278438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrzr4\" (UniqueName: \"kubernetes.io/projected/1d74b1a4-b10a-4477-8515-edb8e5506eb3-kube-api-access-xrzr4\") pod \"authorino-7bcfd8f9cd-b6rjt\" (UID: \"1d74b1a4-b10a-4477-8515-edb8e5506eb3\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b6rjt"
Sep 18 21:14:57.304291 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:57.304258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrzr4\" (UniqueName: \"kubernetes.io/projected/1d74b1a4-b10a-4477-8515-edb8e5506eb3-kube-api-access-xrzr4\") pod \"authorino-7bcfd8f9cd-b6rjt\" (UID: \"1d74b1a4-b10a-4477-8515-edb8e5506eb3\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b6rjt"
Sep 18 21:14:57.342840 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:57.342775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:14:57.395376 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:57.395352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b6rjt"
Sep 18 21:14:57.479514 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:57.479482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5bnx7" event={"ID":"9a0838c7-b955-430e-b22c-257bde994f83","Type":"ContainerStarted","Data":"6e3ab43f1ce091da73075beb725442031d73a9150ab306942ce36b0a00ffabbe"}
Sep 18 21:14:57.668146 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:57.668116    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b6rjt"]
Sep 18 21:14:57.670925 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:14:57.670889    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d74b1a4_b10a_4477_8515_edb8e5506eb3.slice/crio-dd5a3ae95b578edfd8c26b09685612fe096a6a794a804a35542474fcb8740cdf WatchSource:0}: Error finding container dd5a3ae95b578edfd8c26b09685612fe096a6a794a804a35542474fcb8740cdf: Status 404 returned error can't find the container with id dd5a3ae95b578edfd8c26b09685612fe096a6a794a804a35542474fcb8740cdf
Sep 18 21:14:58.485413 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:58.485375    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b6rjt" event={"ID":"1d74b1a4-b10a-4477-8515-edb8e5506eb3","Type":"ContainerStarted","Data":"dd5a3ae95b578edfd8c26b09685612fe096a6a794a804a35542474fcb8740cdf"}
Sep 18 21:14:58.486923 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:58.486893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5bnx7" event={"ID":"9a0838c7-b955-430e-b22c-257bde994f83","Type":"ContainerStarted","Data":"6a1874bde01305de78a578c83312f2116d127f3a1b9db5edc45893d0be064c74"}
Sep 18 21:14:58.487060 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:58.487029    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-5bnx7"
Sep 18 21:14:58.517185 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:14:58.517133    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-5bnx7" podStartSLOduration=2.214935154 podStartE2EDuration="3.51711933s" totalImagesPullingTime="1.302184176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:14:58.514662995 +0000 UTC m=+536.121722545" watchObservedRunningTime="2026-09-18 21:14:58.51711933 +0000 UTC m=+536.124178887"
Sep 18 21:15:00.012306 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:00.012274    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b6rjt"]
Sep 18 21:15:00.499291 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:00.499251    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b6rjt" event={"ID":"1d74b1a4-b10a-4477-8515-edb8e5506eb3","Type":"ContainerStarted","Data":"91b7fe2066b5f0faa155aa72125670ea665c5033c8cc3cc85484abc2ff499698"}
Sep 18 21:15:00.524304 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:00.524252    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-b6rjt" podStartSLOduration=1.665471273 podStartE2EDuration="3.524234976s" totalImagesPullingTime="1.858763703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:15:00.52313824 +0000 UTC m=+538.130197797" watchObservedRunningTime="2026-09-18 21:15:00.524234976 +0000 UTC m=+538.131294534"
Sep 18 21:15:01.504027 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:01.503981    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-b6rjt" podUID="1d74b1a4-b10a-4477-8515-edb8e5506eb3" containerName="authorino" containerID="cri-o://91b7fe2066b5f0faa155aa72125670ea665c5033c8cc3cc85484abc2ff499698" gracePeriod=30
Sep 18 21:15:01.753084 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:01.753052    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b6rjt"
Sep 18 21:15:01.822969 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:01.822934    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrzr4\" (UniqueName: \"kubernetes.io/projected/1d74b1a4-b10a-4477-8515-edb8e5506eb3-kube-api-access-xrzr4\") pod \"1d74b1a4-b10a-4477-8515-edb8e5506eb3\" (UID: \"1d74b1a4-b10a-4477-8515-edb8e5506eb3\") "
Sep 18 21:15:01.825085 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:01.825058    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d74b1a4-b10a-4477-8515-edb8e5506eb3-kube-api-access-xrzr4" (OuterVolumeSpecName: "kube-api-access-xrzr4") pod "1d74b1a4-b10a-4477-8515-edb8e5506eb3" (UID: "1d74b1a4-b10a-4477-8515-edb8e5506eb3"). InnerVolumeSpecName "kube-api-access-xrzr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:15:01.923907 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:01.923869    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrzr4\" (UniqueName: \"kubernetes.io/projected/1d74b1a4-b10a-4477-8515-edb8e5506eb3-kube-api-access-xrzr4\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:15:02.509423 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:02.509390    2579 generic.go:356] "Generic (PLEG): container finished" podID="1d74b1a4-b10a-4477-8515-edb8e5506eb3" containerID="91b7fe2066b5f0faa155aa72125670ea665c5033c8cc3cc85484abc2ff499698" exitCode=0
Sep 18 21:15:02.509865 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:02.509430    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b6rjt"
Sep 18 21:15:02.509865 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:02.509477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b6rjt" event={"ID":"1d74b1a4-b10a-4477-8515-edb8e5506eb3","Type":"ContainerDied","Data":"91b7fe2066b5f0faa155aa72125670ea665c5033c8cc3cc85484abc2ff499698"}
Sep 18 21:15:02.509865 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:02.509516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b6rjt" event={"ID":"1d74b1a4-b10a-4477-8515-edb8e5506eb3","Type":"ContainerDied","Data":"dd5a3ae95b578edfd8c26b09685612fe096a6a794a804a35542474fcb8740cdf"}
Sep 18 21:15:02.509865 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:02.509535    2579 scope.go:117] "RemoveContainer" containerID="91b7fe2066b5f0faa155aa72125670ea665c5033c8cc3cc85484abc2ff499698"
Sep 18 21:15:02.520746 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:02.520719    2579 scope.go:117] "RemoveContainer" containerID="91b7fe2066b5f0faa155aa72125670ea665c5033c8cc3cc85484abc2ff499698"
Sep 18 21:15:02.521343 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:15:02.521318    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b7fe2066b5f0faa155aa72125670ea665c5033c8cc3cc85484abc2ff499698\": container with ID starting with 91b7fe2066b5f0faa155aa72125670ea665c5033c8cc3cc85484abc2ff499698 not found: ID does not exist" containerID="91b7fe2066b5f0faa155aa72125670ea665c5033c8cc3cc85484abc2ff499698"
Sep 18 21:15:02.521442 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:02.521352    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b7fe2066b5f0faa155aa72125670ea665c5033c8cc3cc85484abc2ff499698"} err="failed to get container status \"91b7fe2066b5f0faa155aa72125670ea665c5033c8cc3cc85484abc2ff499698\": rpc error: code = NotFound desc = could not find container \"91b7fe2066b5f0faa155aa72125670ea665c5033c8cc3cc85484abc2ff499698\": container with ID starting with 91b7fe2066b5f0faa155aa72125670ea665c5033c8cc3cc85484abc2ff499698 not found: ID does not exist"
Sep 18 21:15:02.536699 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:02.536670    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b6rjt"]
Sep 18 21:15:02.542594 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:02.542568    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b6rjt"]
Sep 18 21:15:02.927262 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:02.927227    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d74b1a4-b10a-4477-8515-edb8e5506eb3" path="/var/lib/kubelet/pods/1d74b1a4-b10a-4477-8515-edb8e5506eb3/volumes"
Sep 18 21:15:09.493870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:09.493837    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-5bnx7"
Sep 18 21:15:11.732053 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:11.732012    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5bnx7"]
Sep 18 21:15:11.732454 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:11.732221    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-5bnx7" podUID="9a0838c7-b955-430e-b22c-257bde994f83" containerName="limitador" containerID="cri-o://6a1874bde01305de78a578c83312f2116d127f3a1b9db5edc45893d0be064c74" gracePeriod=30
Sep 18 21:15:12.285454 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.285424    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5bnx7"
Sep 18 21:15:12.414490 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.414386    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a0838c7-b955-430e-b22c-257bde994f83-config-file\") pod \"9a0838c7-b955-430e-b22c-257bde994f83\" (UID: \"9a0838c7-b955-430e-b22c-257bde994f83\") "
Sep 18 21:15:12.414490 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.414460    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mt4w\" (UniqueName: \"kubernetes.io/projected/9a0838c7-b955-430e-b22c-257bde994f83-kube-api-access-5mt4w\") pod \"9a0838c7-b955-430e-b22c-257bde994f83\" (UID: \"9a0838c7-b955-430e-b22c-257bde994f83\") "
Sep 18 21:15:12.414786 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.414764    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0838c7-b955-430e-b22c-257bde994f83-config-file" (OuterVolumeSpecName: "config-file") pod "9a0838c7-b955-430e-b22c-257bde994f83" (UID: "9a0838c7-b955-430e-b22c-257bde994f83"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:15:12.416473 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.416445    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0838c7-b955-430e-b22c-257bde994f83-kube-api-access-5mt4w" (OuterVolumeSpecName: "kube-api-access-5mt4w") pod "9a0838c7-b955-430e-b22c-257bde994f83" (UID: "9a0838c7-b955-430e-b22c-257bde994f83"). InnerVolumeSpecName "kube-api-access-5mt4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:15:12.515803 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.515771    2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a0838c7-b955-430e-b22c-257bde994f83-config-file\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:15:12.515803 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.515797    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mt4w\" (UniqueName: \"kubernetes.io/projected/9a0838c7-b955-430e-b22c-257bde994f83-kube-api-access-5mt4w\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:15:12.558162 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.558122    2579 generic.go:356] "Generic (PLEG): container finished" podID="9a0838c7-b955-430e-b22c-257bde994f83" containerID="6a1874bde01305de78a578c83312f2116d127f3a1b9db5edc45893d0be064c74" exitCode=0
Sep 18 21:15:12.558162 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.558164    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5bnx7"
Sep 18 21:15:12.558428 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.558195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5bnx7" event={"ID":"9a0838c7-b955-430e-b22c-257bde994f83","Type":"ContainerDied","Data":"6a1874bde01305de78a578c83312f2116d127f3a1b9db5edc45893d0be064c74"}
Sep 18 21:15:12.558428 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.558241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5bnx7" event={"ID":"9a0838c7-b955-430e-b22c-257bde994f83","Type":"ContainerDied","Data":"6e3ab43f1ce091da73075beb725442031d73a9150ab306942ce36b0a00ffabbe"}
Sep 18 21:15:12.558428 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.558269    2579 scope.go:117] "RemoveContainer" containerID="6a1874bde01305de78a578c83312f2116d127f3a1b9db5edc45893d0be064c74"
Sep 18 21:15:12.567402 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.567386    2579 scope.go:117] "RemoveContainer" containerID="6a1874bde01305de78a578c83312f2116d127f3a1b9db5edc45893d0be064c74"
Sep 18 21:15:12.567673 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:15:12.567635    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a1874bde01305de78a578c83312f2116d127f3a1b9db5edc45893d0be064c74\": container with ID starting with 6a1874bde01305de78a578c83312f2116d127f3a1b9db5edc45893d0be064c74 not found: ID does not exist" containerID="6a1874bde01305de78a578c83312f2116d127f3a1b9db5edc45893d0be064c74"
Sep 18 21:15:12.567728 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.567685    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a1874bde01305de78a578c83312f2116d127f3a1b9db5edc45893d0be064c74"} err="failed to get container status \"6a1874bde01305de78a578c83312f2116d127f3a1b9db5edc45893d0be064c74\": rpc error: code = NotFound desc = could not find container \"6a1874bde01305de78a578c83312f2116d127f3a1b9db5edc45893d0be064c74\": container with ID starting with 6a1874bde01305de78a578c83312f2116d127f3a1b9db5edc45893d0be064c74 not found: ID does not exist"
Sep 18 21:15:12.589293 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.589263    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5bnx7"]
Sep 18 21:15:12.598205 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.598162    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5bnx7"]
Sep 18 21:15:12.921689 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:12.921634    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a0838c7-b955-430e-b22c-257bde994f83" path="/var/lib/kubelet/pods/9a0838c7-b955-430e-b22c-257bde994f83/volumes"
Sep 18 21:15:13.362117 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.362079    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:15:13.464024 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.463983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:15:13.770874 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.770794    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4qccc"]
Sep 18 21:15:13.771186 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.771173    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d74b1a4-b10a-4477-8515-edb8e5506eb3" containerName="authorino"
Sep 18 21:15:13.771228 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.771188    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d74b1a4-b10a-4477-8515-edb8e5506eb3" containerName="authorino"
Sep 18 21:15:13.771228 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.771207    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a0838c7-b955-430e-b22c-257bde994f83" containerName="limitador"
Sep 18 21:15:13.771228 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.771212    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a0838c7-b955-430e-b22c-257bde994f83" containerName="limitador"
Sep 18 21:15:13.771316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.771261    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d74b1a4-b10a-4477-8515-edb8e5506eb3" containerName="authorino"
Sep 18 21:15:13.771316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.771273    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a0838c7-b955-430e-b22c-257bde994f83" containerName="limitador"
Sep 18 21:15:13.775666 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.775619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4qccc"
Sep 18 21:15:13.778658 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.778623    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 18 21:15:13.779143 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.779122    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 21:15:13.779143 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.779138    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 21:15:13.780126 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.780107    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-4dbb7\""
Sep 18 21:15:13.790689 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.790655    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4qccc"]
Sep 18 21:15:13.802407 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.802374    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:15:13.928995 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.928949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hnm\" (UniqueName: \"kubernetes.io/projected/4bafa5e1-5077-47d1-aeff-996533129bcc-kube-api-access-k2hnm\") pod \"postgres-69fb4594cf-4qccc\" (UID: \"4bafa5e1-5077-47d1-aeff-996533129bcc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4qccc"
Sep 18 21:15:13.929403 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:13.929053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4bafa5e1-5077-47d1-aeff-996533129bcc-data\") pod \"postgres-69fb4594cf-4qccc\" (UID: \"4bafa5e1-5077-47d1-aeff-996533129bcc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4qccc"
Sep 18 21:15:14.029764 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:14.029625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hnm\" (UniqueName: \"kubernetes.io/projected/4bafa5e1-5077-47d1-aeff-996533129bcc-kube-api-access-k2hnm\") pod \"postgres-69fb4594cf-4qccc\" (UID: \"4bafa5e1-5077-47d1-aeff-996533129bcc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4qccc"
Sep 18 21:15:14.029955 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:14.029793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4bafa5e1-5077-47d1-aeff-996533129bcc-data\") pod \"postgres-69fb4594cf-4qccc\" (UID: \"4bafa5e1-5077-47d1-aeff-996533129bcc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4qccc"
Sep 18 21:15:14.030227 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:14.030200    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4bafa5e1-5077-47d1-aeff-996533129bcc-data\") pod \"postgres-69fb4594cf-4qccc\" (UID: \"4bafa5e1-5077-47d1-aeff-996533129bcc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4qccc"
Sep 18 21:15:14.038822 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:14.038792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hnm\" (UniqueName: \"kubernetes.io/projected/4bafa5e1-5077-47d1-aeff-996533129bcc-kube-api-access-k2hnm\") pod \"postgres-69fb4594cf-4qccc\" (UID: \"4bafa5e1-5077-47d1-aeff-996533129bcc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4qccc"
Sep 18 21:15:14.086375 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:14.086326    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4qccc"
Sep 18 21:15:14.361667 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:14.361563    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4qccc"]
Sep 18 21:15:14.364398 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:15:14.364365    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bafa5e1_5077_47d1_aeff_996533129bcc.slice/crio-f60ea0dfa04491b08d9a527f453f7e6fe0b258d9c8c8912a86dafed1fb2258cb WatchSource:0}: Error finding container f60ea0dfa04491b08d9a527f453f7e6fe0b258d9c8c8912a86dafed1fb2258cb: Status 404 returned error can't find the container with id f60ea0dfa04491b08d9a527f453f7e6fe0b258d9c8c8912a86dafed1fb2258cb
Sep 18 21:15:14.571160 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:14.571114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4qccc" event={"ID":"4bafa5e1-5077-47d1-aeff-996533129bcc","Type":"ContainerStarted","Data":"f60ea0dfa04491b08d9a527f453f7e6fe0b258d9c8c8912a86dafed1fb2258cb"}
Sep 18 21:15:19.603394 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:19.603358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4qccc" event={"ID":"4bafa5e1-5077-47d1-aeff-996533129bcc","Type":"ContainerStarted","Data":"fd6d0cdac6f469a979bb842c38740d6c78e9ea0bc74ef8003569c7aafdb67f9a"}
Sep 18 21:15:19.603848 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:19.603465    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4qccc"
Sep 18 21:15:19.626127 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:19.626074    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4qccc" podStartSLOduration=1.881602085 podStartE2EDuration="6.626054622s" totalImagesPullingTime="4.744452537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:15:19.624339836 +0000 UTC m=+557.231399408" watchObservedRunningTime="2026-09-18 21:15:19.626054622 +0000 UTC m=+557.233114181"
Sep 18 21:15:25.638115 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:25.638081    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4qccc"
Sep 18 21:15:25.958935 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:25.958845    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:15:26.236723 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.236624    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-jmc9q"]
Sep 18 21:15:26.241845 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.241824    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-jmc9q"
Sep 18 21:15:26.246011 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.245986    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-95fsz\""
Sep 18 21:15:26.246144 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.245986    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 21:15:26.246878 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.246861    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 21:15:26.271560 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.271521    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-jmc9q"]
Sep 18 21:15:26.345792 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.345751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqft9\" (UniqueName: \"kubernetes.io/projected/c038d925-59c8-46cf-af4a-be1dff6f1c5e-kube-api-access-wqft9\") pod \"authorino-557f876f69-jmc9q\" (UID: \"c038d925-59c8-46cf-af4a-be1dff6f1c5e\") " pod="kuadrant-system/authorino-557f876f69-jmc9q"
Sep 18 21:15:26.446563 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.446525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqft9\" (UniqueName: \"kubernetes.io/projected/c038d925-59c8-46cf-af4a-be1dff6f1c5e-kube-api-access-wqft9\") pod \"authorino-557f876f69-jmc9q\" (UID: \"c038d925-59c8-46cf-af4a-be1dff6f1c5e\") " pod="kuadrant-system/authorino-557f876f69-jmc9q"
Sep 18 21:15:26.460568 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.460541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqft9\" (UniqueName: \"kubernetes.io/projected/c038d925-59c8-46cf-af4a-be1dff6f1c5e-kube-api-access-wqft9\") pod \"authorino-557f876f69-jmc9q\" (UID: \"c038d925-59c8-46cf-af4a-be1dff6f1c5e\") " pod="kuadrant-system/authorino-557f876f69-jmc9q"
Sep 18 21:15:26.509999 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.509880    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-jmc9q"]
Sep 18 21:15:26.510240 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.510223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-jmc9q"
Sep 18 21:15:26.543794 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.543752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:15:26.579814 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.579782    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c6f86c85c-zzbdx"]
Sep 18 21:15:26.588897 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.588817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c6f86c85c-zzbdx"
Sep 18 21:15:26.620220 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.620181    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c6f86c85c-zzbdx"]
Sep 18 21:15:26.687353 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.687318    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c6f86c85c-zzbdx"]
Sep 18 21:15:26.687769 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:15:26.687600    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-26lgk], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6c6f86c85c-zzbdx" podUID="bb1f564f-3b6f-45aa-8e29-8e882f9a013e"
Sep 18 21:15:26.749837 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.749798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lgk\" (UniqueName: \"kubernetes.io/projected/bb1f564f-3b6f-45aa-8e29-8e882f9a013e-kube-api-access-26lgk\") pod \"authorino-6c6f86c85c-zzbdx\" (UID: \"bb1f564f-3b6f-45aa-8e29-8e882f9a013e\") " pod="kuadrant-system/authorino-6c6f86c85c-zzbdx"
Sep 18 21:15:26.811990 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.811899    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-jmc9q"]
Sep 18 21:15:26.814822 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:15:26.814782    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc038d925_59c8_46cf_af4a_be1dff6f1c5e.slice/crio-70141812c1814a00f9ab2cc48d70fc56eb902c2e427438627507a7a112e1111d WatchSource:0}: Error finding container 70141812c1814a00f9ab2cc48d70fc56eb902c2e427438627507a7a112e1111d: Status 404 returned error can't find the container with id 70141812c1814a00f9ab2cc48d70fc56eb902c2e427438627507a7a112e1111d
Sep 18 21:15:26.850918 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.850875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lgk\" (UniqueName: \"kubernetes.io/projected/bb1f564f-3b6f-45aa-8e29-8e882f9a013e-kube-api-access-26lgk\") pod \"authorino-6c6f86c85c-zzbdx\" (UID: \"bb1f564f-3b6f-45aa-8e29-8e882f9a013e\") " pod="kuadrant-system/authorino-6c6f86c85c-zzbdx"
Sep 18 21:15:26.862782 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:26.862749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lgk\" (UniqueName: \"kubernetes.io/projected/bb1f564f-3b6f-45aa-8e29-8e882f9a013e-kube-api-access-26lgk\") pod \"authorino-6c6f86c85c-zzbdx\" (UID: \"bb1f564f-3b6f-45aa-8e29-8e882f9a013e\") " pod="kuadrant-system/authorino-6c6f86c85c-zzbdx"
Sep 18 21:15:27.645890 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:27.645799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-jmc9q" event={"ID":"c038d925-59c8-46cf-af4a-be1dff6f1c5e","Type":"ContainerStarted","Data":"30d79c320ec8adc5b49541521a84a92ff27bd6608345adcf76e46a61d22e9cd4"}
Sep 18 21:15:27.645890 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:27.645832    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c6f86c85c-zzbdx"
Sep 18 21:15:27.645890 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:27.645854    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-jmc9q" event={"ID":"c038d925-59c8-46cf-af4a-be1dff6f1c5e","Type":"ContainerStarted","Data":"70141812c1814a00f9ab2cc48d70fc56eb902c2e427438627507a7a112e1111d"}
Sep 18 21:15:27.645890 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:27.645812    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-jmc9q" podUID="c038d925-59c8-46cf-af4a-be1dff6f1c5e" containerName="authorino" containerID="cri-o://30d79c320ec8adc5b49541521a84a92ff27bd6608345adcf76e46a61d22e9cd4" gracePeriod=30
Sep 18 21:15:27.680874 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:27.680844    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c6f86c85c-zzbdx"
Sep 18 21:15:27.693743 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:27.693698    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-jmc9q" podStartSLOduration=1.209013312 podStartE2EDuration="1.69368349s" totalImagesPullingTime="484.670178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:15:27.690205143 +0000 UTC m=+565.297264700" watchObservedRunningTime="2026-09-18 21:15:27.69368349 +0000 UTC m=+565.300743048"
Sep 18 21:15:27.861119 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:27.861087    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26lgk\" (UniqueName: \"kubernetes.io/projected/bb1f564f-3b6f-45aa-8e29-8e882f9a013e-kube-api-access-26lgk\") pod \"bb1f564f-3b6f-45aa-8e29-8e882f9a013e\" (UID: \"bb1f564f-3b6f-45aa-8e29-8e882f9a013e\") "
Sep 18 21:15:27.864919 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:27.864888    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1f564f-3b6f-45aa-8e29-8e882f9a013e-kube-api-access-26lgk" (OuterVolumeSpecName: "kube-api-access-26lgk") pod "bb1f564f-3b6f-45aa-8e29-8e882f9a013e" (UID: "bb1f564f-3b6f-45aa-8e29-8e882f9a013e"). InnerVolumeSpecName "kube-api-access-26lgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:15:27.950469 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:27.950443    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-jmc9q"
Sep 18 21:15:27.962089 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:27.962063    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26lgk\" (UniqueName: \"kubernetes.io/projected/bb1f564f-3b6f-45aa-8e29-8e882f9a013e-kube-api-access-26lgk\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:15:28.063089 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.063052    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqft9\" (UniqueName: \"kubernetes.io/projected/c038d925-59c8-46cf-af4a-be1dff6f1c5e-kube-api-access-wqft9\") pod \"c038d925-59c8-46cf-af4a-be1dff6f1c5e\" (UID: \"c038d925-59c8-46cf-af4a-be1dff6f1c5e\") "
Sep 18 21:15:28.065224 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.065199    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c038d925-59c8-46cf-af4a-be1dff6f1c5e-kube-api-access-wqft9" (OuterVolumeSpecName: "kube-api-access-wqft9") pod "c038d925-59c8-46cf-af4a-be1dff6f1c5e" (UID: "c038d925-59c8-46cf-af4a-be1dff6f1c5e"). InnerVolumeSpecName "kube-api-access-wqft9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:15:28.164953 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.164914    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqft9\" (UniqueName: \"kubernetes.io/projected/c038d925-59c8-46cf-af4a-be1dff6f1c5e-kube-api-access-wqft9\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:15:28.652200 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.652164    2579 generic.go:356] "Generic (PLEG): container finished" podID="c038d925-59c8-46cf-af4a-be1dff6f1c5e" containerID="30d79c320ec8adc5b49541521a84a92ff27bd6608345adcf76e46a61d22e9cd4" exitCode=0
Sep 18 21:15:28.652373 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.652210    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-jmc9q"
Sep 18 21:15:28.652373 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.652248    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-jmc9q" event={"ID":"c038d925-59c8-46cf-af4a-be1dff6f1c5e","Type":"ContainerDied","Data":"30d79c320ec8adc5b49541521a84a92ff27bd6608345adcf76e46a61d22e9cd4"}
Sep 18 21:15:28.652373 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.652285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-jmc9q" event={"ID":"c038d925-59c8-46cf-af4a-be1dff6f1c5e","Type":"ContainerDied","Data":"70141812c1814a00f9ab2cc48d70fc56eb902c2e427438627507a7a112e1111d"}
Sep 18 21:15:28.652373 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.652302    2579 scope.go:117] "RemoveContainer" containerID="30d79c320ec8adc5b49541521a84a92ff27bd6608345adcf76e46a61d22e9cd4"
Sep 18 21:15:28.652519 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.652381    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c6f86c85c-zzbdx"
Sep 18 21:15:28.661373 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.661353    2579 scope.go:117] "RemoveContainer" containerID="30d79c320ec8adc5b49541521a84a92ff27bd6608345adcf76e46a61d22e9cd4"
Sep 18 21:15:28.661716 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:15:28.661690    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d79c320ec8adc5b49541521a84a92ff27bd6608345adcf76e46a61d22e9cd4\": container with ID starting with 30d79c320ec8adc5b49541521a84a92ff27bd6608345adcf76e46a61d22e9cd4 not found: ID does not exist" containerID="30d79c320ec8adc5b49541521a84a92ff27bd6608345adcf76e46a61d22e9cd4"
Sep 18 21:15:28.661800 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.661724    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d79c320ec8adc5b49541521a84a92ff27bd6608345adcf76e46a61d22e9cd4"} err="failed to get container status \"30d79c320ec8adc5b49541521a84a92ff27bd6608345adcf76e46a61d22e9cd4\": rpc error: code = NotFound desc = could not find container \"30d79c320ec8adc5b49541521a84a92ff27bd6608345adcf76e46a61d22e9cd4\": container with ID starting with 30d79c320ec8adc5b49541521a84a92ff27bd6608345adcf76e46a61d22e9cd4 not found: ID does not exist"
Sep 18 21:15:28.697910 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.697875    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c6f86c85c-zzbdx"]
Sep 18 21:15:28.703756 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.703725    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c6f86c85c-zzbdx"]
Sep 18 21:15:28.718212 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.718124    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-jmc9q"]
Sep 18 21:15:28.721544 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.721519    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-jmc9q"]
Sep 18 21:15:28.925863 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.925826    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb1f564f-3b6f-45aa-8e29-8e882f9a013e" path="/var/lib/kubelet/pods/bb1f564f-3b6f-45aa-8e29-8e882f9a013e/volumes"
Sep 18 21:15:28.926058 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:28.926046    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c038d925-59c8-46cf-af4a-be1dff6f1c5e" path="/var/lib/kubelet/pods/c038d925-59c8-46cf-af4a-be1dff6f1c5e/volumes"
Sep 18 21:15:35.430902 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.430858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:15:35.561014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.560971    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b7fb6b9f7-lnpxg"]
Sep 18 21:15:35.561364 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.561350    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c038d925-59c8-46cf-af4a-be1dff6f1c5e" containerName="authorino"
Sep 18 21:15:35.561410 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.561367    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c038d925-59c8-46cf-af4a-be1dff6f1c5e" containerName="authorino"
Sep 18 21:15:35.561448 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.561442    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c038d925-59c8-46cf-af4a-be1dff6f1c5e" containerName="authorino"
Sep 18 21:15:35.566585 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.566560    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:35.574545 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.574519    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 21:15:35.574545 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.574532    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 21:15:35.574767 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.574599    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 21:15:35.574767 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.574607    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ghw2c\""
Sep 18 21:15:35.611827 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.611791    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b7fb6b9f7-lnpxg"]
Sep 18 21:15:35.628950 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.628921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-metrics-certs\") pod \"maas-controller-b7fb6b9f7-lnpxg\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") " pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:35.629082 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.628956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nwlx\" (UniqueName: \"kubernetes.io/projected/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-kube-api-access-9nwlx\") pod \"maas-controller-b7fb6b9f7-lnpxg\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") " pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:35.629137 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.629115    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-webhook-cert\") pod \"maas-controller-b7fb6b9f7-lnpxg\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") " pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:35.679497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.679466    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:15:35.729826 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.729743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-webhook-cert\") pod \"maas-controller-b7fb6b9f7-lnpxg\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") " pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:35.729826 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.729792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-metrics-certs\") pod \"maas-controller-b7fb6b9f7-lnpxg\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") " pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:35.729826 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.729821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nwlx\" (UniqueName: \"kubernetes.io/projected/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-kube-api-access-9nwlx\") pod \"maas-controller-b7fb6b9f7-lnpxg\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") " pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:35.730091 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:15:35.729890    2579 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 18 21:15:35.730091 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:15:35.729955    2579 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 18 21:15:35.730091 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:15:35.729964    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-webhook-cert podName:3f5b0c96-94b1-45cd-b753-fcdbfa5a9522 nodeName:}" failed. No retries permitted until 2026-09-18 21:15:36.229947296 +0000 UTC m=+573.837006832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-webhook-cert") pod "maas-controller-b7fb6b9f7-lnpxg" (UID: "3f5b0c96-94b1-45cd-b753-fcdbfa5a9522") : secret "maas-controller-webhook-cert" not found
Sep 18 21:15:35.730091 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:15:35.730039    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-metrics-certs podName:3f5b0c96-94b1-45cd-b753-fcdbfa5a9522 nodeName:}" failed. No retries permitted until 2026-09-18 21:15:36.230018763 +0000 UTC m=+573.837078313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-metrics-certs") pod "maas-controller-b7fb6b9f7-lnpxg" (UID: "3f5b0c96-94b1-45cd-b753-fcdbfa5a9522") : secret "maas-controller-metrics-tls" not found
Sep 18 21:15:35.751049 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.751019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nwlx\" (UniqueName: \"kubernetes.io/projected/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-kube-api-access-9nwlx\") pod \"maas-controller-b7fb6b9f7-lnpxg\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") " pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:35.839328 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.839292    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-799c76f65c-zjvqk"]
Sep 18 21:15:35.841750 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.841734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:15:35.898152 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.898118    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-799c76f65c-zjvqk"]
Sep 18 21:15:35.906809 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.906774    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:15:35.931235 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.931205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/806deb32-abef-4295-9a73-4693b2682de2-webhook-cert\") pod \"maas-controller-799c76f65c-zjvqk\" (UID: \"806deb32-abef-4295-9a73-4693b2682de2\") " pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:15:35.931500 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.931477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/806deb32-abef-4295-9a73-4693b2682de2-metrics-certs\") pod \"maas-controller-799c76f65c-zjvqk\" (UID: \"806deb32-abef-4295-9a73-4693b2682de2\") " pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:15:35.931596 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:35.931543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6lhg\" (UniqueName: \"kubernetes.io/projected/806deb32-abef-4295-9a73-4693b2682de2-kube-api-access-c6lhg\") pod \"maas-controller-799c76f65c-zjvqk\" (UID: \"806deb32-abef-4295-9a73-4693b2682de2\") " pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:15:36.032900 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.032813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/806deb32-abef-4295-9a73-4693b2682de2-webhook-cert\") pod \"maas-controller-799c76f65c-zjvqk\" (UID: \"806deb32-abef-4295-9a73-4693b2682de2\") " pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:15:36.032900 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.032872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/806deb32-abef-4295-9a73-4693b2682de2-metrics-certs\") pod \"maas-controller-799c76f65c-zjvqk\" (UID: \"806deb32-abef-4295-9a73-4693b2682de2\") " pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:15:36.032900 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.032900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6lhg\" (UniqueName: \"kubernetes.io/projected/806deb32-abef-4295-9a73-4693b2682de2-kube-api-access-c6lhg\") pod \"maas-controller-799c76f65c-zjvqk\" (UID: \"806deb32-abef-4295-9a73-4693b2682de2\") " pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:15:36.035360 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.035331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/806deb32-abef-4295-9a73-4693b2682de2-webhook-cert\") pod \"maas-controller-799c76f65c-zjvqk\" (UID: \"806deb32-abef-4295-9a73-4693b2682de2\") " pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:15:36.035472 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.035425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/806deb32-abef-4295-9a73-4693b2682de2-metrics-certs\") pod \"maas-controller-799c76f65c-zjvqk\" (UID: \"806deb32-abef-4295-9a73-4693b2682de2\") " pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:15:36.047889 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.047863    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6lhg\" (UniqueName: \"kubernetes.io/projected/806deb32-abef-4295-9a73-4693b2682de2-kube-api-access-c6lhg\") pod \"maas-controller-799c76f65c-zjvqk\" (UID: \"806deb32-abef-4295-9a73-4693b2682de2\") " pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:15:36.065308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.065280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:15:36.152802 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.152767    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:15:36.163201 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.163177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:15:36.235005 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.234976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-webhook-cert\") pod \"maas-controller-b7fb6b9f7-lnpxg\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") " pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:36.235160 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.235025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-metrics-certs\") pod \"maas-controller-b7fb6b9f7-lnpxg\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") " pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:36.237300 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.237270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-webhook-cert\") pod \"maas-controller-b7fb6b9f7-lnpxg\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") " pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:36.237425 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.237365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-metrics-certs\") pod \"maas-controller-b7fb6b9f7-lnpxg\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") " pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:36.468052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.468020    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-799c76f65c-zjvqk"]
Sep 18 21:15:36.469016 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:15:36.468978    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806deb32_abef_4295_9a73_4693b2682de2.slice/crio-7b7ba91299f9fcec8e16877492bfcfe92ae3f7d41fe523a3639342b8eb185e9a WatchSource:0}: Error finding container 7b7ba91299f9fcec8e16877492bfcfe92ae3f7d41fe523a3639342b8eb185e9a: Status 404 returned error can't find the container with id 7b7ba91299f9fcec8e16877492bfcfe92ae3f7d41fe523a3639342b8eb185e9a
Sep 18 21:15:36.477015 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.476981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:36.691473 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.691434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799c76f65c-zjvqk" event={"ID":"806deb32-abef-4295-9a73-4693b2682de2","Type":"ContainerStarted","Data":"7b7ba91299f9fcec8e16877492bfcfe92ae3f7d41fe523a3639342b8eb185e9a"}
Sep 18 21:15:36.768345 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:36.768314    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b7fb6b9f7-lnpxg"]
Sep 18 21:15:36.770097 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:15:36.770070    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5b0c96_94b1_45cd_b753_fcdbfa5a9522.slice/crio-a503a7763cc9226a35f8399508980b91b54cb5a6d2af0fe4f5daa57d31db17fd WatchSource:0}: Error finding container a503a7763cc9226a35f8399508980b91b54cb5a6d2af0fe4f5daa57d31db17fd: Status 404 returned error can't find the container with id a503a7763cc9226a35f8399508980b91b54cb5a6d2af0fe4f5daa57d31db17fd
Sep 18 21:15:37.702637 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:37.702592    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg" event={"ID":"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522","Type":"ContainerStarted","Data":"a503a7763cc9226a35f8399508980b91b54cb5a6d2af0fe4f5daa57d31db17fd"}
Sep 18 21:15:39.437170 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:39.437137    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:15:39.615581 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:39.615549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:15:39.713535 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:39.713485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799c76f65c-zjvqk" event={"ID":"806deb32-abef-4295-9a73-4693b2682de2","Type":"ContainerStarted","Data":"45613ec5d9805b3380554ac1c5ed768219f434df65f9714e69e83030916404db"}
Sep 18 21:15:39.713779 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:39.713576    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:15:39.715177 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:39.715148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg" event={"ID":"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522","Type":"ContainerStarted","Data":"7548c3b39f7a528cffef8b5a9c6d635cc7c7baeb520a749e1593fa4b46281888"}
Sep 18 21:15:39.715298 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:39.715203    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:39.735389 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:39.735328    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-799c76f65c-zjvqk" podStartSLOduration=2.000922073 podStartE2EDuration="4.73531009s" totalImagesPullingTime="2.734388017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:15:39.734204466 +0000 UTC m=+577.341264039" watchObservedRunningTime="2026-09-18 21:15:39.73531009 +0000 UTC m=+577.342369648"
Sep 18 21:15:39.767285 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:39.767227    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg" podStartSLOduration=2.324090847 podStartE2EDuration="4.767208296s" totalImagesPullingTime="2.443117449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:15:39.760740104 +0000 UTC m=+577.367799663" watchObservedRunningTime="2026-09-18 21:15:39.767208296 +0000 UTC m=+577.374267886"
Sep 18 21:15:50.721821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:50.721774    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:15:50.722311 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:50.721866    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:50.829834 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:50.829796    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b7fb6b9f7-lnpxg"]
Sep 18 21:15:50.830090 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:50.830059    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg" podUID="3f5b0c96-94b1-45cd-b753-fcdbfa5a9522" containerName="manager" containerID="cri-o://7548c3b39f7a528cffef8b5a9c6d635cc7c7baeb520a749e1593fa4b46281888" gracePeriod=30
Sep 18 21:15:51.077381 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.077352    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:51.178751 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.178718    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nwlx\" (UniqueName: \"kubernetes.io/projected/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-kube-api-access-9nwlx\") pod \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") "
Sep 18 21:15:51.178940 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.178787    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-webhook-cert\") pod \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") "
Sep 18 21:15:51.178940 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.178848    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-metrics-certs\") pod \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\" (UID: \"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522\") "
Sep 18 21:15:51.181125 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.181091    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3f5b0c96-94b1-45cd-b753-fcdbfa5a9522" (UID: "3f5b0c96-94b1-45cd-b753-fcdbfa5a9522"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:15:51.181125 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.181111    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-kube-api-access-9nwlx" (OuterVolumeSpecName: "kube-api-access-9nwlx") pod "3f5b0c96-94b1-45cd-b753-fcdbfa5a9522" (UID: "3f5b0c96-94b1-45cd-b753-fcdbfa5a9522"). InnerVolumeSpecName "kube-api-access-9nwlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:15:51.181315 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.181123    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3f5b0c96-94b1-45cd-b753-fcdbfa5a9522" (UID: "3f5b0c96-94b1-45cd-b753-fcdbfa5a9522"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:15:51.280126 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.280037    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nwlx\" (UniqueName: \"kubernetes.io/projected/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-kube-api-access-9nwlx\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:15:51.280126 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.280074    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-webhook-cert\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:15:51.280126 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.280084    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:15:51.773178 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.773138    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f5b0c96-94b1-45cd-b753-fcdbfa5a9522" containerID="7548c3b39f7a528cffef8b5a9c6d635cc7c7baeb520a749e1593fa4b46281888" exitCode=0
Sep 18 21:15:51.773595 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.773189    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg"
Sep 18 21:15:51.773595 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.773217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg" event={"ID":"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522","Type":"ContainerDied","Data":"7548c3b39f7a528cffef8b5a9c6d635cc7c7baeb520a749e1593fa4b46281888"}
Sep 18 21:15:51.773595 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.773271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7fb6b9f7-lnpxg" event={"ID":"3f5b0c96-94b1-45cd-b753-fcdbfa5a9522","Type":"ContainerDied","Data":"a503a7763cc9226a35f8399508980b91b54cb5a6d2af0fe4f5daa57d31db17fd"}
Sep 18 21:15:51.773595 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.773294    2579 scope.go:117] "RemoveContainer" containerID="7548c3b39f7a528cffef8b5a9c6d635cc7c7baeb520a749e1593fa4b46281888"
Sep 18 21:15:51.782261 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.782246    2579 scope.go:117] "RemoveContainer" containerID="7548c3b39f7a528cffef8b5a9c6d635cc7c7baeb520a749e1593fa4b46281888"
Sep 18 21:15:51.782505 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:15:51.782489    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7548c3b39f7a528cffef8b5a9c6d635cc7c7baeb520a749e1593fa4b46281888\": container with ID starting with 7548c3b39f7a528cffef8b5a9c6d635cc7c7baeb520a749e1593fa4b46281888 not found: ID does not exist" containerID="7548c3b39f7a528cffef8b5a9c6d635cc7c7baeb520a749e1593fa4b46281888"
Sep 18 21:15:51.782553 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.782513    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7548c3b39f7a528cffef8b5a9c6d635cc7c7baeb520a749e1593fa4b46281888"} err="failed to get container status \"7548c3b39f7a528cffef8b5a9c6d635cc7c7baeb520a749e1593fa4b46281888\": rpc error: code = NotFound desc = could not find container \"7548c3b39f7a528cffef8b5a9c6d635cc7c7baeb520a749e1593fa4b46281888\": container with ID starting with 7548c3b39f7a528cffef8b5a9c6d635cc7c7baeb520a749e1593fa4b46281888 not found: ID does not exist"
Sep 18 21:15:51.800991 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.800962    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b7fb6b9f7-lnpxg"]
Sep 18 21:15:51.805904 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:51.805867    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b7fb6b9f7-lnpxg"]
Sep 18 21:15:52.921598 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:15:52.921561    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f5b0c96-94b1-45cd-b753-fcdbfa5a9522" path="/var/lib/kubelet/pods/3f5b0c96-94b1-45cd-b753-fcdbfa5a9522/volumes"
Sep 18 21:16:02.822087 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:02.822053    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:16:02.823266 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:02.823244    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:16:02.826004 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:02.825983    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:16:02.827134 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:02.827117    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:16:11.171469 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:11.171430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:16:11.466070 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:11.465979    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:16:12.355541 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.355501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:16:12.469811 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.469774    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:16:12.488328 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.488295    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"]
Sep 18 21:16:12.488700 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.488687    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5b0c96-94b1-45cd-b753-fcdbfa5a9522" containerName="manager"
Sep 18 21:16:12.488746 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.488702    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5b0c96-94b1-45cd-b753-fcdbfa5a9522" containerName="manager"
Sep 18 21:16:12.488780 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.488770    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f5b0c96-94b1-45cd-b753-fcdbfa5a9522" containerName="manager"
Sep 18 21:16:12.493298 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.493281    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:12.496198 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.496178    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 21:16:12.497456 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.497440    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 21:16:12.497525 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.497461    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dcnql\""
Sep 18 21:16:12.519878 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.519852    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"]
Sep 18 21:16:12.575228 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.575199    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:16:12.580386 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.580366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-maas-api-tls\") pod \"maas-api-5598d8c9fb-lh827\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:12.580459 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.580396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-metrics-certs\") pod \"maas-api-5598d8c9fb-lh827\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:12.580495 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.580465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3160642c-34c1-4164-b837-f3a13ad1f3e5-openshift-service-ca\") pod \"maas-api-5598d8c9fb-lh827\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:12.580495 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.580489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8kz4\" (UniqueName: \"kubernetes.io/projected/3160642c-34c1-4164-b837-f3a13ad1f3e5-kube-api-access-t8kz4\") pod \"maas-api-5598d8c9fb-lh827\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:12.680985 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.680909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8kz4\" (UniqueName: \"kubernetes.io/projected/3160642c-34c1-4164-b837-f3a13ad1f3e5-kube-api-access-t8kz4\") pod \"maas-api-5598d8c9fb-lh827\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:12.680985 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.680972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-maas-api-tls\") pod \"maas-api-5598d8c9fb-lh827\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:12.681254 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.680998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-metrics-certs\") pod \"maas-api-5598d8c9fb-lh827\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:12.681254 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.681067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3160642c-34c1-4164-b837-f3a13ad1f3e5-openshift-service-ca\") pod \"maas-api-5598d8c9fb-lh827\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:12.681254 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:16:12.681085    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 18 21:16:12.681254 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:16:12.681168    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-maas-api-tls podName:3160642c-34c1-4164-b837-f3a13ad1f3e5 nodeName:}" failed. No retries permitted until 2026-09-18 21:16:13.181147831 +0000 UTC m=+610.788207370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-maas-api-tls") pod "maas-api-5598d8c9fb-lh827" (UID: "3160642c-34c1-4164-b837-f3a13ad1f3e5") : secret "maas-api-serving-cert" not found
Sep 18 21:16:12.681720 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.681702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3160642c-34c1-4164-b837-f3a13ad1f3e5-openshift-service-ca\") pod \"maas-api-5598d8c9fb-lh827\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:12.683429 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.683409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-metrics-certs\") pod \"maas-api-5598d8c9fb-lh827\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:12.691684 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.691663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8kz4\" (UniqueName: \"kubernetes.io/projected/3160642c-34c1-4164-b837-f3a13ad1f3e5-kube-api-access-t8kz4\") pod \"maas-api-5598d8c9fb-lh827\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:12.754993 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.754955    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:16:12.925056 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:12.924966    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:16:13.019022 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.018921    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:16:13.184683 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.184625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-maas-api-tls\") pod \"maas-api-5598d8c9fb-lh827\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:13.187129 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.187102    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-maas-api-tls\") pod \"maas-api-5598d8c9fb-lh827\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") " pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:13.227925 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.227889    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-9kv57"]
Sep 18 21:16:13.230749 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.230733    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-9kv57"
Sep 18 21:16:13.234174 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.234149    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 18 21:16:13.234281 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.234188    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6kd5q\""
Sep 18 21:16:13.250566 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.250539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-9kv57"]
Sep 18 21:16:13.285768 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.285739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b4xf\" (UniqueName: \"kubernetes.io/projected/0392321b-d271-4c1c-beea-5c2e7d5802db-kube-api-access-7b4xf\") pod \"payload-processing-5b54b7fbb7-9kv57\" (UID: \"0392321b-d271-4c1c-beea-5c2e7d5802db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9kv57"
Sep 18 21:16:13.285972 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.285886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0392321b-d271-4c1c-beea-5c2e7d5802db-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-9kv57\" (UID: \"0392321b-d271-4c1c-beea-5c2e7d5802db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9kv57"
Sep 18 21:16:13.387137 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.387100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0392321b-d271-4c1c-beea-5c2e7d5802db-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-9kv57\" (UID: \"0392321b-d271-4c1c-beea-5c2e7d5802db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9kv57"
Sep 18 21:16:13.387629 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.387156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b4xf\" (UniqueName: \"kubernetes.io/projected/0392321b-d271-4c1c-beea-5c2e7d5802db-kube-api-access-7b4xf\") pod \"payload-processing-5b54b7fbb7-9kv57\" (UID: \"0392321b-d271-4c1c-beea-5c2e7d5802db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9kv57"
Sep 18 21:16:13.387998 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.387972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0392321b-d271-4c1c-beea-5c2e7d5802db-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-9kv57\" (UID: \"0392321b-d271-4c1c-beea-5c2e7d5802db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9kv57"
Sep 18 21:16:13.396248 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.396216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b4xf\" (UniqueName: \"kubernetes.io/projected/0392321b-d271-4c1c-beea-5c2e7d5802db-kube-api-access-7b4xf\") pod \"payload-processing-5b54b7fbb7-9kv57\" (UID: \"0392321b-d271-4c1c-beea-5c2e7d5802db\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9kv57"
Sep 18 21:16:13.403953 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.403931    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:13.541600 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.541510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-9kv57"
Sep 18 21:16:13.700561 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.700503    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"]
Sep 18 21:16:13.703297 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:16:13.703267    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3160642c_34c1_4164_b837_f3a13ad1f3e5.slice/crio-64f885bb3af3868f825bf71273296abdb033f01dc0f68fc3e124dd99fe1c0702 WatchSource:0}: Error finding container 64f885bb3af3868f825bf71273296abdb033f01dc0f68fc3e124dd99fe1c0702: Status 404 returned error can't find the container with id 64f885bb3af3868f825bf71273296abdb033f01dc0f68fc3e124dd99fe1c0702
Sep 18 21:16:13.846639 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.846557    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-9kv57"]
Sep 18 21:16:13.860705 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:16:13.860671    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0392321b_d271_4c1c_beea_5c2e7d5802db.slice/crio-21001accd272c4b5207526b24cbb705cffe2b6929e47426b9e1de8069923c2ea WatchSource:0}: Error finding container 21001accd272c4b5207526b24cbb705cffe2b6929e47426b9e1de8069923c2ea: Status 404 returned error can't find the container with id 21001accd272c4b5207526b24cbb705cffe2b6929e47426b9e1de8069923c2ea
Sep 18 21:16:13.883577 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.883547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-9kv57" event={"ID":"0392321b-d271-4c1c-beea-5c2e7d5802db","Type":"ContainerStarted","Data":"21001accd272c4b5207526b24cbb705cffe2b6929e47426b9e1de8069923c2ea"}
Sep 18 21:16:13.884794 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:13.884772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827" event={"ID":"3160642c-34c1-4164-b837-f3a13ad1f3e5","Type":"ContainerStarted","Data":"64f885bb3af3868f825bf71273296abdb033f01dc0f68fc3e124dd99fe1c0702"}
Sep 18 21:16:15.903365 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:15.903324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-9kv57" event={"ID":"0392321b-d271-4c1c-beea-5c2e7d5802db","Type":"ContainerStarted","Data":"7aa6224c735d52c74bc6e2c10c4d7556bf73ffa817f18cea92f01dca7077f2fa"}
Sep 18 21:16:15.904041 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:15.903421    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-9kv57"
Sep 18 21:16:15.905480 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:15.905450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827" event={"ID":"3160642c-34c1-4164-b837-f3a13ad1f3e5","Type":"ContainerStarted","Data":"0935946ecbd0bd9ee775563dede549e9b9066778424c6f008dfb18c67b58e938"}
Sep 18 21:16:15.905605 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:15.905578    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:15.923613 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:15.923509    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-9kv57" podStartSLOduration=1.119323479 podStartE2EDuration="2.923490304s" totalImagesPullingTime="1.804166825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:16:15.922907506 +0000 UTC m=+613.529967065" watchObservedRunningTime="2026-09-18 21:16:15.923490304 +0000 UTC m=+613.530549863"
Sep 18 21:16:15.947906 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:15.947851    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827" podStartSLOduration=1.987904736 podStartE2EDuration="3.947835505s" totalImagesPullingTime="1.959930769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:16:15.947207846 +0000 UTC m=+613.554267405" watchObservedRunningTime="2026-09-18 21:16:15.947835505 +0000 UTC m=+613.554895063"
Sep 18 21:16:21.915957 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:21.915923    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:16:22.944216 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:22.944175    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:16:23.046388 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:23.046355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:16:26.913231 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:26.913151    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-9kv57"
Sep 18 21:16:35.105369 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.105333    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"]
Sep 18 21:16:35.108943 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.108922    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.112884 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.112858    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gv5wt\""
Sep 18 21:16:35.112884 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.112878    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 21:16:35.112884 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.112884    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 21:16:35.113181 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.113166    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 18 21:16:35.119871 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.119850    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"]
Sep 18 21:16:35.134955 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.134914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:16:35.190032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.190000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/384a741b-99f9-45c6-8596-f927061235ad-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.190032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.190036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/384a741b-99f9-45c6-8596-f927061235ad-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.190218 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.190058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/384a741b-99f9-45c6-8596-f927061235ad-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.190218 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.190172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjcjn\" (UniqueName: \"kubernetes.io/projected/384a741b-99f9-45c6-8596-f927061235ad-kube-api-access-fjcjn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.190295 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.190227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/384a741b-99f9-45c6-8596-f927061235ad-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.190295 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.190248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/384a741b-99f9-45c6-8596-f927061235ad-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.291683 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.291605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/384a741b-99f9-45c6-8596-f927061235ad-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.291683 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.291665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/384a741b-99f9-45c6-8596-f927061235ad-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.291929 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.291722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/384a741b-99f9-45c6-8596-f927061235ad-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.291929 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.291740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/384a741b-99f9-45c6-8596-f927061235ad-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.291929 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.291773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/384a741b-99f9-45c6-8596-f927061235ad-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.291929 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.291825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjcjn\" (UniqueName: \"kubernetes.io/projected/384a741b-99f9-45c6-8596-f927061235ad-kube-api-access-fjcjn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.292143 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.292065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/384a741b-99f9-45c6-8596-f927061235ad-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.292143 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.292115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/384a741b-99f9-45c6-8596-f927061235ad-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.292265 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.292150    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/384a741b-99f9-45c6-8596-f927061235ad-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.294019 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.294002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/384a741b-99f9-45c6-8596-f927061235ad-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.294174 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.294155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/384a741b-99f9-45c6-8596-f927061235ad-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.301891 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.301867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjcjn\" (UniqueName: \"kubernetes.io/projected/384a741b-99f9-45c6-8596-f927061235ad-kube-api-access-fjcjn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp\" (UID: \"384a741b-99f9-45c6-8596-f927061235ad\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.419773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.419694    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:35.701779 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.701753    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"]
Sep 18 21:16:35.704508 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:16:35.704475    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384a741b_99f9_45c6_8596_f927061235ad.slice/crio-805b35a4cbc2730486f9151e131a5b5d6122e050b87c58549638ce3a85922af9 WatchSource:0}: Error finding container 805b35a4cbc2730486f9151e131a5b5d6122e050b87c58549638ce3a85922af9: Status 404 returned error can't find the container with id 805b35a4cbc2730486f9151e131a5b5d6122e050b87c58549638ce3a85922af9
Sep 18 21:16:35.706141 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:35.706118    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:16:36.006976 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:36.006892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp" event={"ID":"384a741b-99f9-45c6-8596-f927061235ad","Type":"ContainerStarted","Data":"805b35a4cbc2730486f9151e131a5b5d6122e050b87c58549638ce3a85922af9"}
Sep 18 21:16:37.013588 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:37.013559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp" event={"ID":"384a741b-99f9-45c6-8596-f927061235ad","Type":"ContainerStarted","Data":"990e64c1dedaa7f7d6242f465e72e9c936a43d36f9631b282f803ed0f4a9fa14"}
Sep 18 21:16:37.038693 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:37.038609    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp" podStartSLOduration=0.943510952 podStartE2EDuration="2.038586936s" totalImagesPullingTime="1.095075984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:16:37.038207452 +0000 UTC m=+634.645267006" watchObservedRunningTime="2026-09-18 21:16:37.038586936 +0000 UTC m=+634.645646492"
Sep 18 21:16:42.490780 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:42.490746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:16:45.420750 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:45.420710    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:45.421156 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:45.420782    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:45.433348 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:45.433325    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:46.065676 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:46.065595    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp"
Sep 18 21:16:58.956956 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:16:58.956912    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:17:03.267708 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:03.267668    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:17:03.348622 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:03.348574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:17:08.356657 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:08.356613    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:17:08.448554 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:08.448515    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:17:10.732682 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:10.732634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:17:21.517423 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.517390    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"]
Sep 18 21:17:21.522825 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.522803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.526161 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.526137    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 18 21:17:21.536235 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.536206    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"]
Sep 18 21:17:21.560280 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.560246    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:17:21.709316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.709277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d809256-f651-48e0-b3ba-f62340bed671-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.709316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.709318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d809256-f651-48e0-b3ba-f62340bed671-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.709527 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.709447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d809256-f651-48e0-b3ba-f62340bed671-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.709527 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.709475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d809256-f651-48e0-b3ba-f62340bed671-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.709527 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.709495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d809256-f651-48e0-b3ba-f62340bed671-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.709630 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.709566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85w8p\" (UniqueName: \"kubernetes.io/projected/2d809256-f651-48e0-b3ba-f62340bed671-kube-api-access-85w8p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.810929 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.810889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d809256-f651-48e0-b3ba-f62340bed671-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.810929 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.810931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d809256-f651-48e0-b3ba-f62340bed671-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.811171 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.810953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d809256-f651-48e0-b3ba-f62340bed671-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.811171 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.811011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85w8p\" (UniqueName: \"kubernetes.io/projected/2d809256-f651-48e0-b3ba-f62340bed671-kube-api-access-85w8p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.811171 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.811047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d809256-f651-48e0-b3ba-f62340bed671-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.811171 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.811074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d809256-f651-48e0-b3ba-f62340bed671-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.811440 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.811405    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d809256-f651-48e0-b3ba-f62340bed671-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.811519 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.811494    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d809256-f651-48e0-b3ba-f62340bed671-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.811580 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.811502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d809256-f651-48e0-b3ba-f62340bed671-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.813341 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.813319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d809256-f651-48e0-b3ba-f62340bed671-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.813597 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.813577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d809256-f651-48e0-b3ba-f62340bed671-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.825106 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.825078    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85w8p\" (UniqueName: \"kubernetes.io/projected/2d809256-f651-48e0-b3ba-f62340bed671-kube-api-access-85w8p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92\" (UID: \"2d809256-f651-48e0-b3ba-f62340bed671\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:21.833025 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:21.832978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:22.113224 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:22.113193    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"]
Sep 18 21:17:22.115474 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:17:22.115443    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d809256_f651_48e0_b3ba_f62340bed671.slice/crio-1ab17969a7ce16505d2258d6e742842a872ac78ecd468311e785cbb77585f8f4 WatchSource:0}: Error finding container 1ab17969a7ce16505d2258d6e742842a872ac78ecd468311e785cbb77585f8f4: Status 404 returned error can't find the container with id 1ab17969a7ce16505d2258d6e742842a872ac78ecd468311e785cbb77585f8f4
Sep 18 21:17:22.227700 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:22.227658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92" event={"ID":"2d809256-f651-48e0-b3ba-f62340bed671","Type":"ContainerStarted","Data":"1ab17969a7ce16505d2258d6e742842a872ac78ecd468311e785cbb77585f8f4"}
Sep 18 21:17:23.233578 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:23.233541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92" event={"ID":"2d809256-f651-48e0-b3ba-f62340bed671","Type":"ContainerStarted","Data":"2ae0f00884f6d6ff821612c5b8cc379d812efb123ccd89e9d92d699671b5b79f"}
Sep 18 21:17:23.255412 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:23.255370    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92" podStartSLOduration=2.089751716 podStartE2EDuration="2.255355779s" totalImagesPullingTime="165.604063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:17:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:17:23.253162934 +0000 UTC m=+680.860222491" watchObservedRunningTime="2026-09-18 21:17:23.255355779 +0000 UTC m=+680.862415337"
Sep 18 21:17:31.299521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:31.299484    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:17:31.833297 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:31.833265    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:31.833461 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:31.833327    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:31.845717 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:31.845690    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:32.289879 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:32.289855    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92"
Sep 18 21:17:37.987043 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:37.987008    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:17:52.269339 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:52.269300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:17:57.434789 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.434707    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"]
Sep 18 21:17:57.438473 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.438447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.442624 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.442601    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 18 21:17:57.452921 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.452892    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"]
Sep 18 21:17:57.482955 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.482905    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:17:57.513163 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.513125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63cdd729-6d75-4d94-bab1-0c1c5c7068be-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.513332 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.513175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l28vb\" (UniqueName: \"kubernetes.io/projected/63cdd729-6d75-4d94-bab1-0c1c5c7068be-kube-api-access-l28vb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.513332 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.513213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63cdd729-6d75-4d94-bab1-0c1c5c7068be-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.513332 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.513281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63cdd729-6d75-4d94-bab1-0c1c5c7068be-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.513441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.513327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63cdd729-6d75-4d94-bab1-0c1c5c7068be-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.513441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.513376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63cdd729-6d75-4d94-bab1-0c1c5c7068be-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.614803 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.614766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63cdd729-6d75-4d94-bab1-0c1c5c7068be-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.614979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.614809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63cdd729-6d75-4d94-bab1-0c1c5c7068be-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.614979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.614855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63cdd729-6d75-4d94-bab1-0c1c5c7068be-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.614979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.614918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63cdd729-6d75-4d94-bab1-0c1c5c7068be-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.614979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.614959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28vb\" (UniqueName: \"kubernetes.io/projected/63cdd729-6d75-4d94-bab1-0c1c5c7068be-kube-api-access-l28vb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.615181 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.614990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63cdd729-6d75-4d94-bab1-0c1c5c7068be-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.615375 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.615349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63cdd729-6d75-4d94-bab1-0c1c5c7068be-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.615455 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.615406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63cdd729-6d75-4d94-bab1-0c1c5c7068be-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.615497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.615450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63cdd729-6d75-4d94-bab1-0c1c5c7068be-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.617229 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.617206    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63cdd729-6d75-4d94-bab1-0c1c5c7068be-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.617375 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.617358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63cdd729-6d75-4d94-bab1-0c1c5c7068be-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.623900 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.623877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l28vb\" (UniqueName: \"kubernetes.io/projected/63cdd729-6d75-4d94-bab1-0c1c5c7068be-kube-api-access-l28vb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn\" (UID: \"63cdd729-6d75-4d94-bab1-0c1c5c7068be\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:57.748513 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:57.748419    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:17:58.051192 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:58.051151    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"]
Sep 18 21:17:58.053565 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:17:58.053538    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63cdd729_6d75_4d94_bab1_0c1c5c7068be.slice/crio-4b7a8e63ff64d3d250dd78a8ee272869232e7a8ba0ea55753ec68d61cab87f02 WatchSource:0}: Error finding container 4b7a8e63ff64d3d250dd78a8ee272869232e7a8ba0ea55753ec68d61cab87f02: Status 404 returned error can't find the container with id 4b7a8e63ff64d3d250dd78a8ee272869232e7a8ba0ea55753ec68d61cab87f02
Sep 18 21:17:58.410229 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:58.410145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn" event={"ID":"63cdd729-6d75-4d94-bab1-0c1c5c7068be","Type":"ContainerStarted","Data":"c1c77ff04d032827d3035692296dc1824c7249fe2793916f957106a92569564a"}
Sep 18 21:17:58.410229 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:17:58.410187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn" event={"ID":"63cdd729-6d75-4d94-bab1-0c1c5c7068be","Type":"ContainerStarted","Data":"4b7a8e63ff64d3d250dd78a8ee272869232e7a8ba0ea55753ec68d61cab87f02"}
Sep 18 21:18:07.749507 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:18:07.749463    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:18:07.749991 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:18:07.749523    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:18:07.762154 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:18:07.762119    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:18:07.786075 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:18:07.786025    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn" podStartSLOduration=10.611233508 podStartE2EDuration="10.786010154s" totalImagesPullingTime="174.776646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:17:58.452189597 +0000 UTC m=+716.059249146" watchObservedRunningTime="2026-09-18 21:18:07.786010154 +0000 UTC m=+725.393069711"
Sep 18 21:18:08.467605 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:18:08.467571    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn"
Sep 18 21:19:19.277525 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:19.277483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:19.712337 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:19.712304    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:20.758485 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:20.758451    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:21.216576 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:21.216540    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:36.958541 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:36.958456    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:37.113313 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.113279    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:37.452069 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.452031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:37.603207 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.603173    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:37.692609 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.692565    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"]
Sep 18 21:19:37.698003 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.697973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.701362 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.701338    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a7d035-openshift-default-dockercfg-l5lbc\""
Sep 18 21:19:37.701624 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.701603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 21:19:37.717043 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.717018    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"]
Sep 18 21:19:37.742572 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.742546    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:37.772173 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.772147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istiod-ca-cert\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.772264 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.772183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-workload-certs\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.772264 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.772249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-token\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.772341 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.772281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-credential-socket\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.772341 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.772305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-envoy\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.772341 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.772323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-data\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.772436 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.772351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-workload-socket\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.772436 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.772381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-podinfo\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.772436 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.772411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgnr9\" (UniqueName: \"kubernetes.io/projected/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-kube-api-access-dgnr9\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.865262 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.865235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:37.873400 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.873375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istiod-ca-cert\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.873514 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.873407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-workload-certs\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.873514 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.873438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-token\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.873514 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.873458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-credential-socket\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.873514 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.873478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-envoy\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.873514 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.873496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-data\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.873514 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.873516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-workload-socket\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.873869 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.873533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-podinfo\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.873869 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.873597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgnr9\" (UniqueName: \"kubernetes.io/projected/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-kube-api-access-dgnr9\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.873869 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.873825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-workload-certs\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.874033 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.873982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-workload-socket\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.874033 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.873981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-data\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.874138 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.874042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-credential-socket\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.874247 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.874226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istiod-ca-cert\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.875773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.875755    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-envoy\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.876009 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.875990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-podinfo\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.885471 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.885445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-token\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.885631 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.885610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgnr9\" (UniqueName: \"kubernetes.io/projected/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-kube-api-access-dgnr9\") pod \"e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") " pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:37.981216 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:37.981155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:38.010219 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:38.010196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:38.076797 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:38.076732    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:38.290333 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:38.290120    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"]
Sep 18 21:19:38.292477 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:19:38.292452    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6638fb_3ff8_451d_8ab7_63ea4f2de3ce.slice/crio-c76e2b40606d0dcda375c2e6c55427fe4001e3ea3304993798b60019d1635c6a WatchSource:0}: Error finding container c76e2b40606d0dcda375c2e6c55427fe4001e3ea3304993798b60019d1635c6a: Status 404 returned error can't find the container with id c76e2b40606d0dcda375c2e6c55427fe4001e3ea3304993798b60019d1635c6a
Sep 18 21:19:38.364461 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:38.364433    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:38.487580 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:38.487527    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:38.841754 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:38.841716    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:38.911491 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:38.911460    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44" event={"ID":"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce","Type":"ContainerStarted","Data":"c76e2b40606d0dcda375c2e6c55427fe4001e3ea3304993798b60019d1635c6a"}
Sep 18 21:19:40.603006 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:40.602949    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:40.731678 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:40.731617    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:40.829085 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:40.829049    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:41.264245 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:41.264202    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:19:41.264363 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:41.264295    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:19:41.264363 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:41.264341    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:19:41.932096 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:41.932057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44" event={"ID":"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce","Type":"ContainerStarted","Data":"7840e9992736df733e47a107905755093ada63a0143c987579ffa4c7a5a57936"}
Sep 18 21:19:42.010999 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.010964    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:42.016176 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.016148    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:42.017115 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.017063    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44" podStartSLOduration=2.047538448 podStartE2EDuration="5.017044964s" totalImagesPullingTime="2.969506516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:19:42.016293271 +0000 UTC m=+819.623352831" watchObservedRunningTime="2026-09-18 21:19:42.017044964 +0000 UTC m=+819.624104524"
Sep 18 21:19:42.274542 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.274469    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"]
Sep 18 21:19:42.277795 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.277778    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.287988 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.287967    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-116f55-serving-cert\""
Sep 18 21:19:42.309284 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.309259    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"]
Sep 18 21:19:42.417778 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.417745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fc27ce-2b89-43c8-97b7-e2f2807637e2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-116f55-68d7469686-k4fzg\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.417948 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.417809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rjnx\" (UniqueName: \"kubernetes.io/projected/14fc27ce-2b89-43c8-97b7-e2f2807637e2-kube-api-access-9rjnx\") pod \"maas-api-e2e-inf-a-116f55-68d7469686-k4fzg\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.418021 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.417941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14fc27ce-2b89-43c8-97b7-e2f2807637e2-metrics-certs\") pod \"maas-api-e2e-inf-a-116f55-68d7469686-k4fzg\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.418021 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.417995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14fc27ce-2b89-43c8-97b7-e2f2807637e2-maas-api-tls\") pod \"maas-api-e2e-inf-a-116f55-68d7469686-k4fzg\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.418269 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.418249    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:42.519171 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.519146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14fc27ce-2b89-43c8-97b7-e2f2807637e2-metrics-certs\") pod \"maas-api-e2e-inf-a-116f55-68d7469686-k4fzg\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.519303 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.519191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14fc27ce-2b89-43c8-97b7-e2f2807637e2-maas-api-tls\") pod \"maas-api-e2e-inf-a-116f55-68d7469686-k4fzg\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.519370 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.519308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fc27ce-2b89-43c8-97b7-e2f2807637e2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-116f55-68d7469686-k4fzg\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.519370 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.519347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rjnx\" (UniqueName: \"kubernetes.io/projected/14fc27ce-2b89-43c8-97b7-e2f2807637e2-kube-api-access-9rjnx\") pod \"maas-api-e2e-inf-a-116f55-68d7469686-k4fzg\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.520014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.519993    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fc27ce-2b89-43c8-97b7-e2f2807637e2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-116f55-68d7469686-k4fzg\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.521597 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.521574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14fc27ce-2b89-43c8-97b7-e2f2807637e2-metrics-certs\") pod \"maas-api-e2e-inf-a-116f55-68d7469686-k4fzg\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.521699 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.521612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14fc27ce-2b89-43c8-97b7-e2f2807637e2-maas-api-tls\") pod \"maas-api-e2e-inf-a-116f55-68d7469686-k4fzg\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.536934 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.536906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rjnx\" (UniqueName: \"kubernetes.io/projected/14fc27ce-2b89-43c8-97b7-e2f2807637e2-kube-api-access-9rjnx\") pod \"maas-api-e2e-inf-a-116f55-68d7469686-k4fzg\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.588849 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.588824    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:42.751199 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.751167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:42.904798 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.904728    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"]
Sep 18 21:19:42.907955 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:19:42.907928    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fc27ce_2b89_43c8_97b7_e2f2807637e2.slice/crio-3ee9eb97be2e5dc0f92b8041a32ef3aacb5a367c3aa9b6806e2fcc51756eb4d7 WatchSource:0}: Error finding container 3ee9eb97be2e5dc0f92b8041a32ef3aacb5a367c3aa9b6806e2fcc51756eb4d7: Status 404 returned error can't find the container with id 3ee9eb97be2e5dc0f92b8041a32ef3aacb5a367c3aa9b6806e2fcc51756eb4d7
Sep 18 21:19:42.937973 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.937945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg" event={"ID":"14fc27ce-2b89-43c8-97b7-e2f2807637e2","Type":"ContainerStarted","Data":"3ee9eb97be2e5dc0f92b8041a32ef3aacb5a367c3aa9b6806e2fcc51756eb4d7"}
Sep 18 21:19:42.938252 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.938180    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:42.938995 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:42.938977    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:19:43.946115 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:43.946074    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg" event={"ID":"14fc27ce-2b89-43c8-97b7-e2f2807637e2","Type":"ContainerStarted","Data":"9a0346f30581cce72a00f9bc13d7c8d1a111187b1f5281fb5fa5860dc84354a3"}
Sep 18 21:19:43.989137 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:43.988985    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg" podStartSLOduration=1.6030946209999999 podStartE2EDuration="1.988964844s" totalImagesPullingTime="385.870223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:19:43.988120502 +0000 UTC m=+821.595180061" watchObservedRunningTime="2026-09-18 21:19:43.988964844 +0000 UTC m=+821.596024403"
Sep 18 21:19:43.989618 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:43.989584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:44.193871 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:44.193842    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:44.664539 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:44.664506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:44.951298 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:44.951164    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:45.031149 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.031113    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:45.317359 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.317326    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:45.421487 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.421449    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"]
Sep 18 21:19:45.425073 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.425052    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:45.440184 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.440165    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-6ad49b-serving-cert\""
Sep 18 21:19:45.463260 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.463238    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"]
Sep 18 21:19:45.549337 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.549313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-metrics-certs\") pod \"maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:45.549459 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.549360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28b5k\" (UniqueName: \"kubernetes.io/projected/f76ac646-0715-400d-ad3d-fcb28f93ef9e-kube-api-access-28b5k\") pod \"maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:45.549528 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.549451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ac646-0715-400d-ad3d-fcb28f93ef9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:45.549589 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.549569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:45.576151 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.576089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:45.650958 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.650924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:45.651096 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.650969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-metrics-certs\") pod \"maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:45.651096 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.651008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28b5k\" (UniqueName: \"kubernetes.io/projected/f76ac646-0715-400d-ad3d-fcb28f93ef9e-kube-api-access-28b5k\") pod \"maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:45.651096 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.651042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ac646-0715-400d-ad3d-fcb28f93ef9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:45.651096 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:19:45.651066    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-serving-cert: secret "maas-api-e2e-worker-w4-6ad49b-serving-cert" not found
Sep 18 21:19:45.651297 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:19:45.651128    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-maas-api-tls podName:f76ac646-0715-400d-ad3d-fcb28f93ef9e nodeName:}" failed. No retries permitted until 2026-09-18 21:19:46.151113335 +0000 UTC m=+823.758172871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-maas-api-tls") pod "maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf" (UID: "f76ac646-0715-400d-ad3d-fcb28f93ef9e") : secret "maas-api-e2e-worker-w4-6ad49b-serving-cert" not found
Sep 18 21:19:45.651667 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.651628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ac646-0715-400d-ad3d-fcb28f93ef9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:45.653213 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.653193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-metrics-certs\") pod \"maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:45.706970 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.706952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28b5k\" (UniqueName: \"kubernetes.io/projected/f76ac646-0715-400d-ad3d-fcb28f93ef9e-kube-api-access-28b5k\") pod \"maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:45.825533 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:45.825509    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:46.154707 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:46.154676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:46.157121 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:46.157099    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:46.157272 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:46.157248    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:46.335369 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:46.335342    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:46.682175 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:46.682140    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"]
Sep 18 21:19:46.684869 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:19:46.684842    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76ac646_0715_400d_ad3d_fcb28f93ef9e.slice/crio-20f82bd81839871c06bf935eea127fb181f41f9840e5faf08e482e1bfeeaa0d5 WatchSource:0}: Error finding container 20f82bd81839871c06bf935eea127fb181f41f9840e5faf08e482e1bfeeaa0d5: Status 404 returned error can't find the container with id 20f82bd81839871c06bf935eea127fb181f41f9840e5faf08e482e1bfeeaa0d5
Sep 18 21:19:46.962289 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:46.962204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf" event={"ID":"f76ac646-0715-400d-ad3d-fcb28f93ef9e","Type":"ContainerStarted","Data":"20f82bd81839871c06bf935eea127fb181f41f9840e5faf08e482e1bfeeaa0d5"}
Sep 18 21:19:47.631943 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:47.631908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:47.817609 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:47.817580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:47.969927 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:47.969849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf" event={"ID":"f76ac646-0715-400d-ad3d-fcb28f93ef9e","Type":"ContainerStarted","Data":"26063e6c5c0ea884b678880b723f6cea370b4b4c13aebe986af522cc1741b1e6"}
Sep 18 21:19:47.970076 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:47.969968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:48.009283 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.009242    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf" podStartSLOduration=2.651290754 podStartE2EDuration="3.009225098s" totalImagesPullingTime="357.934344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:19:48.008962137 +0000 UTC m=+825.616021695" watchObservedRunningTime="2026-09-18 21:19:48.009225098 +0000 UTC m=+825.616284657"
Sep 18 21:19:48.521419 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.521385    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"]
Sep 18 21:19:48.525001 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.524981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:48.527770 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.527749    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-a7d035-serving-cert\""
Sep 18 21:19:48.538464 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.538441    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"]
Sep 18 21:19:48.586586 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.586549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:48.683588 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.683555    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:48.686849 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.686826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c26ef2b-d211-402f-87ce-d635a79eb990-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:48.686923 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.686862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c26ef2b-d211-402f-87ce-d635a79eb990-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:48.686923 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.686880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c26ef2b-d211-402f-87ce-d635a79eb990-metrics-certs\") pod \"maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:48.687032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.686985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mdpc\" (UniqueName: \"kubernetes.io/projected/2c26ef2b-d211-402f-87ce-d635a79eb990-kube-api-access-6mdpc\") pod \"maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:48.787587 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.787566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c26ef2b-d211-402f-87ce-d635a79eb990-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:48.787735 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.787604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c26ef2b-d211-402f-87ce-d635a79eb990-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:48.787735 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.787631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c26ef2b-d211-402f-87ce-d635a79eb990-metrics-certs\") pod \"maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:48.787735 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.787710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mdpc\" (UniqueName: \"kubernetes.io/projected/2c26ef2b-d211-402f-87ce-d635a79eb990-kube-api-access-6mdpc\") pod \"maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:48.788337 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.788313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c26ef2b-d211-402f-87ce-d635a79eb990-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:48.790108 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.790084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c26ef2b-d211-402f-87ce-d635a79eb990-metrics-certs\") pod \"maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:48.790191 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.790125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c26ef2b-d211-402f-87ce-d635a79eb990-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:48.794014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.793988    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:48.799794 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.799773    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mdpc\" (UniqueName: \"kubernetes.io/projected/2c26ef2b-d211-402f-87ce-d635a79eb990-kube-api-access-6mdpc\") pod \"maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:48.835743 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:48.835720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:49.140719 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:49.140696    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"]
Sep 18 21:19:49.141867 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:19:49.141842    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c26ef2b_d211_402f_87ce_d635a79eb990.slice/crio-85e331cde51936f181ba3585a7969c4f5e31763a92c4c2dd35b0c13c34882e26 WatchSource:0}: Error finding container 85e331cde51936f181ba3585a7969c4f5e31763a92c4c2dd35b0c13c34882e26: Status 404 returned error can't find the container with id 85e331cde51936f181ba3585a7969c4f5e31763a92c4c2dd35b0c13c34882e26
Sep 18 21:19:49.148130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:49.148103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:49.302983 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:49.302952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:49.454068 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:49.453976    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:49.648236 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:49.648208    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:49.981520 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:49.981482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj" event={"ID":"2c26ef2b-d211-402f-87ce-d635a79eb990","Type":"ContainerStarted","Data":"41feb33af0cc9af89dc3994b20a56bdb4703974210732917e5457f8af1a8e8cc"}
Sep 18 21:19:49.981925 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:49.981546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj" event={"ID":"2c26ef2b-d211-402f-87ce-d635a79eb990","Type":"ContainerStarted","Data":"85e331cde51936f181ba3585a7969c4f5e31763a92c4c2dd35b0c13c34882e26"}
Sep 18 21:19:49.981925 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:49.981658    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:50.016014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:50.015967    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj" podStartSLOduration=1.6176490829999999 podStartE2EDuration="2.015950328s" totalImagesPullingTime="398.301245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:19:50.013342375 +0000 UTC m=+827.620401934" watchObservedRunningTime="2026-09-18 21:19:50.015950328 +0000 UTC m=+827.623009929"
Sep 18 21:19:50.438473 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:50.438440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:50.960834 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:50.960809    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:19:51.554938 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:51.554899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:52.556250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:52.556216    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:52.840591 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:52.840516    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:53.051775 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:53.051740    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:53.335205 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:53.335165    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:53.769502 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:53.769430    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"]
Sep 18 21:19:53.773285 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:53.773263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:53.780433 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:53.780409    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-21df59-serving-cert\""
Sep 18 21:19:53.937055 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:53.937024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c8533bd-22ba-4af4-853c-37c5d34feb9d-maas-api-tls\") pod \"maas-api-e2e-autores-21df59-797b78bc5-f2j9l\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:53.937190 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:53.937070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c8533bd-22ba-4af4-853c-37c5d34feb9d-metrics-certs\") pod \"maas-api-e2e-autores-21df59-797b78bc5-f2j9l\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:53.937190 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:53.937161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8533bd-22ba-4af4-853c-37c5d34feb9d-openshift-service-ca\") pod \"maas-api-e2e-autores-21df59-797b78bc5-f2j9l\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:53.937276 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:53.937221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkb8l\" (UniqueName: \"kubernetes.io/projected/2c8533bd-22ba-4af4-853c-37c5d34feb9d-kube-api-access-gkb8l\") pod \"maas-api-e2e-autores-21df59-797b78bc5-f2j9l\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:53.960746 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:53.960721    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:53.981442 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:53.981407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"]
Sep 18 21:19:53.983544 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:53.983523    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:19:54.038195 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:54.038169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8533bd-22ba-4af4-853c-37c5d34feb9d-openshift-service-ca\") pod \"maas-api-e2e-autores-21df59-797b78bc5-f2j9l\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:54.038334 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:54.038258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkb8l\" (UniqueName: \"kubernetes.io/projected/2c8533bd-22ba-4af4-853c-37c5d34feb9d-kube-api-access-gkb8l\") pod \"maas-api-e2e-autores-21df59-797b78bc5-f2j9l\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:54.038334 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:54.038315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c8533bd-22ba-4af4-853c-37c5d34feb9d-maas-api-tls\") pod \"maas-api-e2e-autores-21df59-797b78bc5-f2j9l\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:54.038429 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:54.038377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c8533bd-22ba-4af4-853c-37c5d34feb9d-metrics-certs\") pod \"maas-api-e2e-autores-21df59-797b78bc5-f2j9l\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:54.038896 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:54.038871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8533bd-22ba-4af4-853c-37c5d34feb9d-openshift-service-ca\") pod \"maas-api-e2e-autores-21df59-797b78bc5-f2j9l\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:54.040888 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:54.040866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c8533bd-22ba-4af4-853c-37c5d34feb9d-maas-api-tls\") pod \"maas-api-e2e-autores-21df59-797b78bc5-f2j9l\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:54.041016 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:54.040899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c8533bd-22ba-4af4-853c-37c5d34feb9d-metrics-certs\") pod \"maas-api-e2e-autores-21df59-797b78bc5-f2j9l\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:54.096677 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:54.096636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkb8l\" (UniqueName: \"kubernetes.io/projected/2c8533bd-22ba-4af4-853c-37c5d34feb9d-kube-api-access-gkb8l\") pod \"maas-api-e2e-autores-21df59-797b78bc5-f2j9l\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:54.338365 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:54.338295    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:54.387142 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:54.387114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:54.682023 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:54.681999    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"]
Sep 18 21:19:54.683850 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:19:54.683826    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8533bd_22ba_4af4_853c_37c5d34feb9d.slice/crio-4899377a5a87e651d5bf748c430e9b9509a8b8e7f18695acdf1674e5814afe12 WatchSource:0}: Error finding container 4899377a5a87e651d5bf748c430e9b9509a8b8e7f18695acdf1674e5814afe12: Status 404 returned error can't find the container with id 4899377a5a87e651d5bf748c430e9b9509a8b8e7f18695acdf1674e5814afe12
Sep 18 21:19:54.781563 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:54.781534    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:55.007911 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:55.007829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l" event={"ID":"2c8533bd-22ba-4af4-853c-37c5d34feb9d","Type":"ContainerStarted","Data":"4899377a5a87e651d5bf748c430e9b9509a8b8e7f18695acdf1674e5814afe12"}
Sep 18 21:19:55.485363 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:55.485330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:55.888620 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:55.888586    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:55.991625 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:55.991597    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:19:56.014872 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.014835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l" event={"ID":"2c8533bd-22ba-4af4-853c-37c5d34feb9d","Type":"ContainerStarted","Data":"5a036307e1ad64a80c38023f5082986ed2820b0346fde01e119381f9d4418547"}
Sep 18 21:19:56.015025 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.014885    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:19:56.097554 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.097498    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l" podStartSLOduration=2.679141923 podStartE2EDuration="3.097478869s" totalImagesPullingTime="418.336946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:19:56.093464301 +0000 UTC m=+833.700523861" watchObservedRunningTime="2026-09-18 21:19:56.097478869 +0000 UTC m=+833.704538428"
Sep 18 21:19:56.102242 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.102213    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:56.251727 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.245157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:56.310673 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.310629    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"]
Sep 18 21:19:56.314298 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.314279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"
Sep 18 21:19:56.317517 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.317491    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-21df59\""
Sep 18 21:19:56.317632 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.317519    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-21df59-dockercfg-r7jkw\""
Sep 18 21:19:56.338666 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.336368    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"]
Sep 18 21:19:56.464238 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.464195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf6eceb-c370-46a5-9726-7677e951ce5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879\" (UID: \"6bf6eceb-c370-46a5-9726-7677e951ce5d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"
Sep 18 21:19:56.464413 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.464266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6j2h\" (UniqueName: \"kubernetes.io/projected/6bf6eceb-c370-46a5-9726-7677e951ce5d-kube-api-access-b6j2h\") pod \"payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879\" (UID: \"6bf6eceb-c370-46a5-9726-7677e951ce5d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"
Sep 18 21:19:56.489519 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.489487    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:56.565969 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.565938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6j2h\" (UniqueName: \"kubernetes.io/projected/6bf6eceb-c370-46a5-9726-7677e951ce5d-kube-api-access-b6j2h\") pod \"payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879\" (UID: \"6bf6eceb-c370-46a5-9726-7677e951ce5d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"
Sep 18 21:19:56.566198 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.566083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf6eceb-c370-46a5-9726-7677e951ce5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879\" (UID: \"6bf6eceb-c370-46a5-9726-7677e951ce5d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"
Sep 18 21:19:56.567058 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.567036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf6eceb-c370-46a5-9726-7677e951ce5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879\" (UID: \"6bf6eceb-c370-46a5-9726-7677e951ce5d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"
Sep 18 21:19:56.697324 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.697285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6j2h\" (UniqueName: \"kubernetes.io/projected/6bf6eceb-c370-46a5-9726-7677e951ce5d-kube-api-access-b6j2h\") pod \"payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879\" (UID: \"6bf6eceb-c370-46a5-9726-7677e951ce5d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"
Sep 18 21:19:56.929681 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:56.929586    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"
Sep 18 21:19:57.049709 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:57.047476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:57.247753 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:57.247727    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"]
Sep 18 21:19:57.251053 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:19:57.251015    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf6eceb_c370_46a5_9726_7677e951ce5d.slice/crio-d623000685264724cd7329f0cb15b7440cc174ab021209260285d8ff63011b66 WatchSource:0}: Error finding container d623000685264724cd7329f0cb15b7440cc174ab021209260285d8ff63011b66: Status 404 returned error can't find the container with id d623000685264724cd7329f0cb15b7440cc174ab021209260285d8ff63011b66
Sep 18 21:19:57.293979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:57.293954    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:57.763352 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:57.763321    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:58.056474 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:58.056434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879" event={"ID":"6bf6eceb-c370-46a5-9726-7677e951ce5d","Type":"ContainerStarted","Data":"8d06bcd8349b9bf1c4502a34c4274003a369b46a4fe78a3fa634e4466102c470"}
Sep 18 21:19:58.056935 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:58.056485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879" event={"ID":"6bf6eceb-c370-46a5-9726-7677e951ce5d","Type":"ContainerStarted","Data":"d623000685264724cd7329f0cb15b7440cc174ab021209260285d8ff63011b66"}
Sep 18 21:19:58.056935 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:58.056512    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"
Sep 18 21:19:58.084583 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:58.084537    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879" podStartSLOduration=2.084522284 podStartE2EDuration="2.084522284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:19:58.082812369 +0000 UTC m=+835.689871926" watchObservedRunningTime="2026-09-18 21:19:58.084522284 +0000 UTC m=+835.691581871"
Sep 18 21:19:59.406861 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:59.406827    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:19:59.948107 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:19:59.948068    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:00.104807 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:00.104771    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:00.243777 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:00.243679    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:00.538260 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:00.538228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:01.006463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:01.006374    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:01.127416 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:01.127372    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:01.328806 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:01.328762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:01.520934 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:01.520854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:02.054554 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:02.054524    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:20:02.880124 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:02.880074    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:04.037197 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.037162    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"]
Sep 18 21:20:04.040207 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.040184    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.055616 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.055593    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ef01a1fd-serving-cert\""
Sep 18 21:20:04.067694 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.067671    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"]
Sep 18 21:20:04.092259 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.092185    2579 generic.go:356] "Generic (PLEG): container finished" podID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerID="7840e9992736df733e47a107905755093ada63a0143c987579ffa4c7a5a57936" exitCode=0
Sep 18 21:20:04.092259 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.092234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44" event={"ID":"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce","Type":"ContainerDied","Data":"7840e9992736df733e47a107905755093ada63a0143c987579ffa4c7a5a57936"}
Sep 18 21:20:04.092905 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.092883    2579 scope.go:117] "RemoveContainer" containerID="7840e9992736df733e47a107905755093ada63a0143c987579ffa4c7a5a57936"
Sep 18 21:20:04.097569 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.096739    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:20:04.097569 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.096989    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:20:04.097569 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.097103    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:20:04.138446 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.138418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwh5b\" (UniqueName: \"kubernetes.io/projected/63099c05-ab68-411d-b9e4-755615c39f55-kube-api-access-rwh5b\") pod \"maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.138540 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.138490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63099c05-ab68-411d-b9e4-755615c39f55-openshift-service-ca\") pod \"maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.138540 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.138520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63099c05-ab68-411d-b9e4-755615c39f55-maas-api-tls\") pod \"maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.138626 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.138565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63099c05-ab68-411d-b9e4-755615c39f55-metrics-certs\") pod \"maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.153525 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.153501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:04.240254 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.240221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwh5b\" (UniqueName: \"kubernetes.io/projected/63099c05-ab68-411d-b9e4-755615c39f55-kube-api-access-rwh5b\") pod \"maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.240428 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.240319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63099c05-ab68-411d-b9e4-755615c39f55-openshift-service-ca\") pod \"maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.240428 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.240363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63099c05-ab68-411d-b9e4-755615c39f55-maas-api-tls\") pod \"maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.240428 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.240419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63099c05-ab68-411d-b9e4-755615c39f55-metrics-certs\") pod \"maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.241062 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.241030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63099c05-ab68-411d-b9e4-755615c39f55-openshift-service-ca\") pod \"maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.242795 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.242771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63099c05-ab68-411d-b9e4-755615c39f55-metrics-certs\") pod \"maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.243100 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.243080    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63099c05-ab68-411d-b9e4-755615c39f55-maas-api-tls\") pod \"maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.268743 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.268712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwh5b\" (UniqueName: \"kubernetes.io/projected/63099c05-ab68-411d-b9e4-755615c39f55-kube-api-access-rwh5b\") pod \"maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.349663 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.349566    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:04.368606 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.368575    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:04.662740 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.662713    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"]
Sep 18 21:20:04.665611 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:20:04.665584    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63099c05_ab68_411d_b9e4_755615c39f55.slice/crio-534c7d4eca11248e976559191a3a53d5de7b6f1992567d2b99a842973a95d10b WatchSource:0}: Error finding container 534c7d4eca11248e976559191a3a53d5de7b6f1992567d2b99a842973a95d10b: Status 404 returned error can't find the container with id 534c7d4eca11248e976559191a3a53d5de7b6f1992567d2b99a842973a95d10b
Sep 18 21:20:04.687922 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.687891    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:04.869257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:04.869223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:05.100436 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:05.100387    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74" event={"ID":"63099c05-ab68-411d-b9e4-755615c39f55","Type":"ContainerStarted","Data":"6e9d9c053a53ed63eb4f190d61e71427387a54f58ee971b761eb9f0a7c9b057b"}
Sep 18 21:20:05.100783 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:05.100445    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:05.100783 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:05.100468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74" event={"ID":"63099c05-ab68-411d-b9e4-755615c39f55","Type":"ContainerStarted","Data":"534c7d4eca11248e976559191a3a53d5de7b6f1992567d2b99a842973a95d10b"}
Sep 18 21:20:05.102690 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:05.102668    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44" event={"ID":"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce","Type":"ContainerStarted","Data":"0be8c71a11c5201bb5a870c02a9eea58ee87682ed2a84c15b3e4c500a890637b"}
Sep 18 21:20:05.159127 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:05.159077    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74" podStartSLOduration=1.790610161 podStartE2EDuration="2.159059144s" totalImagesPullingTime="368.448983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:20:05.152217988 +0000 UTC m=+842.759277545" watchObservedRunningTime="2026-09-18 21:20:05.159059144 +0000 UTC m=+842.766118702"
Sep 18 21:20:05.269472 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:05.269434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:05.668595 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:05.668557    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:06.011148 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:06.011070    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:20:06.011289 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:06.011212    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 18 21:20:06.011289 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:06.011261    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44" podUID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 18 21:20:06.234930 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:06.234800    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:06.569354 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:06.569315    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:06.772728 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:06.772694    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:07.010683 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:07.010589    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 18 21:20:07.010683 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:07.010673    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44" podUID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 18 21:20:07.058200 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:07.058169    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:08.011040 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.011010    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 18 21:20:08.011397 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.011064    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44" podUID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 18 21:20:08.353936 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.353906    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"]
Sep 18 21:20:08.357463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.357439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:08.361035 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.361011    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1f86a8-serving-cert\""
Sep 18 21:20:08.400736 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.400705    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"]
Sep 18 21:20:08.482491 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.482451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-metrics-certs\") pod \"maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:08.482696 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.482511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44824cba-e6f7-401d-a981-23c66c97d895-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:08.482696 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.482584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x966t\" (UniqueName: \"kubernetes.io/projected/44824cba-e6f7-401d-a981-23c66c97d895-kube-api-access-x966t\") pod \"maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:08.482696 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.482675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:08.559152 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.559118    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:08.583884 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.583850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x966t\" (UniqueName: \"kubernetes.io/projected/44824cba-e6f7-401d-a981-23c66c97d895-kube-api-access-x966t\") pod \"maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:08.584036 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.583908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:08.584036 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.583962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-metrics-certs\") pod \"maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:08.584150 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:20:08.584054    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-serving-cert: secret "maas-api-e2e-inf-b-1f86a8-serving-cert" not found
Sep 18 21:20:08.584150 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.584094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44824cba-e6f7-401d-a981-23c66c97d895-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:08.584150 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:20:08.584108    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-maas-api-tls podName:44824cba-e6f7-401d-a981-23c66c97d895 nodeName:}" failed. No retries permitted until 2026-09-18 21:20:09.084093793 +0000 UTC m=+846.691153330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-maas-api-tls") pod "maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9" (UID: "44824cba-e6f7-401d-a981-23c66c97d895") : secret "maas-api-e2e-inf-b-1f86a8-serving-cert" not found
Sep 18 21:20:08.584725 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.584708    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44824cba-e6f7-401d-a981-23c66c97d895-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:08.586288 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.586269    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-metrics-certs\") pod \"maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:08.604936 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.604866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x966t\" (UniqueName: \"kubernetes.io/projected/44824cba-e6f7-401d-a981-23c66c97d895-kube-api-access-x966t\") pod \"maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:08.762361 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.762320    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:08.920941 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:08.920831    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:09.010718 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:09.010685    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 18 21:20:09.010882 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:09.010737    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44" podUID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 18 21:20:09.064740 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:09.064713    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"
Sep 18 21:20:09.088335 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:09.088306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:09.090639 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:09.090618    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:09.268979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:09.268887    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:09.589783 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:09.589755    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"]
Sep 18 21:20:09.590872 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:20:09.590847    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44824cba_e6f7_401d_a981_23c66c97d895.slice/crio-d9ccd6feb9e0202c8a8a41dfb79f18988c03728d7a7522441a7ca01433f8e0f4 WatchSource:0}: Error finding container d9ccd6feb9e0202c8a8a41dfb79f18988c03728d7a7522441a7ca01433f8e0f4: Status 404 returned error can't find the container with id d9ccd6feb9e0202c8a8a41dfb79f18988c03728d7a7522441a7ca01433f8e0f4
Sep 18 21:20:10.010766 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:10.010671    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Sep 18 21:20:10.010766 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:10.010737    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44" podUID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Sep 18 21:20:10.137308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:10.136638    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9" event={"ID":"44824cba-e6f7-401d-a981-23c66c97d895","Type":"ContainerStarted","Data":"00ae26f9f9f21e7af25e2cfde362c247c12e47f08e32c313e7c0ed861e8f3864"}
Sep 18 21:20:10.137308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:10.136757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9" event={"ID":"44824cba-e6f7-401d-a981-23c66c97d895","Type":"ContainerStarted","Data":"d9ccd6feb9e0202c8a8a41dfb79f18988c03728d7a7522441a7ca01433f8e0f4"}
Sep 18 21:20:10.137308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:10.137007    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:10.141184 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:10.141147    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:10.551477 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:10.551434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:11.077442 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:11.077410    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:20:11.077788 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:11.077763    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:20:11.079223 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:11.079203    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:20:11.113600 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:11.113570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:11.145679 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:11.145614    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9" podStartSLOduration=2.703147222 podStartE2EDuration="3.145597966s" totalImagesPullingTime="442.450744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:20:10.241805701 +0000 UTC m=+847.848865259" watchObservedRunningTime="2026-09-18 21:20:11.145597966 +0000 UTC m=+848.752657578"
Sep 18 21:20:11.290123 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:11.290073    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:12.165119 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:12.165085    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:12.983665 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:12.983614    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:14.252269 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:14.252237    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:14.625375 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:14.625330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:14.718239 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:14.718210    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:15.444298 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:15.444255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:16.148263 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:16.148236    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:20:17.259563 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:17.259526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:18.690358 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:18.690311    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:19.892785 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:19.892747    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:21.371272 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:21.371235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:22.583879 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:22.583847    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:23.422261 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:23.422227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:24.364550 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:24.364497    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:25.117956 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:25.117919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:25.267155 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:25.267123    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:26.456426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:26.456378    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:27.378547 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.378514    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"]
Sep 18 21:20:27.378797 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.378772    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74" podUID="63099c05-ab68-411d-b9e4-755615c39f55" containerName="maas-api" containerID="cri-o://6e9d9c053a53ed63eb4f190d61e71427387a54f58ee971b761eb9f0a7c9b057b" gracePeriod=30
Sep 18 21:20:27.416094 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.416061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:27.543361 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.543328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:27.625792 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.625770    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:27.682493 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.682414    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:27.768098 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.768076    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63099c05-ab68-411d-b9e4-755615c39f55-metrics-certs\") pod \"63099c05-ab68-411d-b9e4-755615c39f55\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") "
Sep 18 21:20:27.768238 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.768106    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63099c05-ab68-411d-b9e4-755615c39f55-maas-api-tls\") pod \"63099c05-ab68-411d-b9e4-755615c39f55\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") "
Sep 18 21:20:27.768238 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.768210    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwh5b\" (UniqueName: \"kubernetes.io/projected/63099c05-ab68-411d-b9e4-755615c39f55-kube-api-access-rwh5b\") pod \"63099c05-ab68-411d-b9e4-755615c39f55\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") "
Sep 18 21:20:27.768350 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.768331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63099c05-ab68-411d-b9e4-755615c39f55-openshift-service-ca\") pod \"63099c05-ab68-411d-b9e4-755615c39f55\" (UID: \"63099c05-ab68-411d-b9e4-755615c39f55\") "
Sep 18 21:20:27.768727 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.768697    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63099c05-ab68-411d-b9e4-755615c39f55-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63099c05-ab68-411d-b9e4-755615c39f55" (UID: "63099c05-ab68-411d-b9e4-755615c39f55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:20:27.770398 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.770375    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63099c05-ab68-411d-b9e4-755615c39f55-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "63099c05-ab68-411d-b9e4-755615c39f55" (UID: "63099c05-ab68-411d-b9e4-755615c39f55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:20:27.770620 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.770601    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63099c05-ab68-411d-b9e4-755615c39f55-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63099c05-ab68-411d-b9e4-755615c39f55" (UID: "63099c05-ab68-411d-b9e4-755615c39f55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:20:27.770717 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.770605    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63099c05-ab68-411d-b9e4-755615c39f55-kube-api-access-rwh5b" (OuterVolumeSpecName: "kube-api-access-rwh5b") pod "63099c05-ab68-411d-b9e4-755615c39f55" (UID: "63099c05-ab68-411d-b9e4-755615c39f55"). InnerVolumeSpecName "kube-api-access-rwh5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:20:27.870015 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.869985    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwh5b\" (UniqueName: \"kubernetes.io/projected/63099c05-ab68-411d-b9e4-755615c39f55-kube-api-access-rwh5b\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:20:27.870015 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.870009    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63099c05-ab68-411d-b9e4-755615c39f55-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:20:27.870015 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.870018    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63099c05-ab68-411d-b9e4-755615c39f55-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:20:27.870220 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.870026    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63099c05-ab68-411d-b9e4-755615c39f55-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:20:27.943285 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:27.943224    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:28.139636 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.139606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:28.240336 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.240248    2579 generic.go:356] "Generic (PLEG): container finished" podID="63099c05-ab68-411d-b9e4-755615c39f55" containerID="6e9d9c053a53ed63eb4f190d61e71427387a54f58ee971b761eb9f0a7c9b057b" exitCode=0
Sep 18 21:20:28.240336 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.240299    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"
Sep 18 21:20:28.240528 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.240343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74" event={"ID":"63099c05-ab68-411d-b9e4-755615c39f55","Type":"ContainerDied","Data":"6e9d9c053a53ed63eb4f190d61e71427387a54f58ee971b761eb9f0a7c9b057b"}
Sep 18 21:20:28.240528 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.240407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74" event={"ID":"63099c05-ab68-411d-b9e4-755615c39f55","Type":"ContainerDied","Data":"534c7d4eca11248e976559191a3a53d5de7b6f1992567d2b99a842973a95d10b"}
Sep 18 21:20:28.240528 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.240440    2579 scope.go:117] "RemoveContainer" containerID="6e9d9c053a53ed63eb4f190d61e71427387a54f58ee971b761eb9f0a7c9b057b"
Sep 18 21:20:28.251566 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.251548    2579 scope.go:117] "RemoveContainer" containerID="6e9d9c053a53ed63eb4f190d61e71427387a54f58ee971b761eb9f0a7c9b057b"
Sep 18 21:20:28.251878 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:20:28.251858    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9d9c053a53ed63eb4f190d61e71427387a54f58ee971b761eb9f0a7c9b057b\": container with ID starting with 6e9d9c053a53ed63eb4f190d61e71427387a54f58ee971b761eb9f0a7c9b057b not found: ID does not exist" containerID="6e9d9c053a53ed63eb4f190d61e71427387a54f58ee971b761eb9f0a7c9b057b"
Sep 18 21:20:28.251963 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.251887    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9d9c053a53ed63eb4f190d61e71427387a54f58ee971b761eb9f0a7c9b057b"} err="failed to get container status \"6e9d9c053a53ed63eb4f190d61e71427387a54f58ee971b761eb9f0a7c9b057b\": rpc error: code = NotFound desc = could not find container \"6e9d9c053a53ed63eb4f190d61e71427387a54f58ee971b761eb9f0a7c9b057b\": container with ID starting with 6e9d9c053a53ed63eb4f190d61e71427387a54f58ee971b761eb9f0a7c9b057b not found: ID does not exist"
Sep 18 21:20:28.280531 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.280507    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"]
Sep 18 21:20:28.284628 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.284605    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ef01a1fd-846c8696b5-7jg74"]
Sep 18 21:20:28.401296 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.401266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:28.630845 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.630818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:28.878782 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.878745    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:28.921981 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:28.921921    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63099c05-ab68-411d-b9e4-755615c39f55" path="/var/lib/kubelet/pods/63099c05-ab68-411d-b9e4-755615c39f55/volumes"
Sep 18 21:20:29.457327 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:29.457294    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:29.653529 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:29.653496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:30.620892 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:30.620864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:32.064781 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:32.064748    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:32.953217 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:32.953188    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:33.062244 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:33.062198    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:33.162970 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:33.162942    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:33.939921 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:33.939890    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:34.921300 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:34.921265    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:35.664396 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.664366    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"]
Sep 18 21:20:35.664930 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.664909    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63099c05-ab68-411d-b9e4-755615c39f55" containerName="maas-api"
Sep 18 21:20:35.665010 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.664934    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="63099c05-ab68-411d-b9e4-755615c39f55" containerName="maas-api"
Sep 18 21:20:35.665063 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.665051    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="63099c05-ab68-411d-b9e4-755615c39f55" containerName="maas-api"
Sep 18 21:20:35.667596 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.667576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:35.671609 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.671584    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7e01a72f-serving-cert\""
Sep 18 21:20:35.686445 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.686422    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"]
Sep 18 21:20:35.750261 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.750236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:35.837793 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.837764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-openshift-service-ca\") pod \"maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:35.837897 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.837799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6mfb\" (UniqueName: \"kubernetes.io/projected/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-kube-api-access-f6mfb\") pod \"maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:35.837897 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.837880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-metrics-certs\") pod \"maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:35.837970 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.837905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-maas-api-tls\") pod \"maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:35.843141 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.843117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:35.939172 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.939114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-openshift-service-ca\") pod \"maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:35.939172 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.939146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6mfb\" (UniqueName: \"kubernetes.io/projected/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-kube-api-access-f6mfb\") pod \"maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:35.939483 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.939201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-metrics-certs\") pod \"maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:35.939483 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.939226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-maas-api-tls\") pod \"maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:35.939827 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.939806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-openshift-service-ca\") pod \"maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:35.941516 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.941487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-metrics-certs\") pod \"maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:35.941671 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.941628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-maas-api-tls\") pod \"maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:35.950369 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.950348    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6mfb\" (UniqueName: \"kubernetes.io/projected/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-kube-api-access-f6mfb\") pod \"maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:35.980830 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:35.980805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:36.280989 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:36.280909    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"]
Sep 18 21:20:36.284585 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:20:36.284543    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790f6cb7_41ea_40f7_b955_b3e78ee4a37e.slice/crio-834e11db548a509469611061671124b0ba5866c0bae9e7f6c288710c0daf84f3 WatchSource:0}: Error finding container 834e11db548a509469611061671124b0ba5866c0bae9e7f6c288710c0daf84f3: Status 404 returned error can't find the container with id 834e11db548a509469611061671124b0ba5866c0bae9e7f6c288710c0daf84f3
Sep 18 21:20:36.725504 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:36.725470    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:36.997247 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:36.997176    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:37.297007 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:37.296971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx" event={"ID":"790f6cb7-41ea-40f7-b955-b3e78ee4a37e","Type":"ContainerStarted","Data":"e74ee334bfcf26ff07c9433ec240d5d9e6f7f22ff3eb066db46d685f7bac742f"}
Sep 18 21:20:37.297195 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:37.297060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx" event={"ID":"790f6cb7-41ea-40f7-b955-b3e78ee4a37e","Type":"ContainerStarted","Data":"834e11db548a509469611061671124b0ba5866c0bae9e7f6c288710c0daf84f3"}
Sep 18 21:20:37.297195 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:37.297088    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:37.360048 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:37.360004    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx" podStartSLOduration=1.8122057759999999 podStartE2EDuration="2.3599863s" totalImagesPullingTime="547.780524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:20:37.358493419 +0000 UTC m=+874.965552977" watchObservedRunningTime="2026-09-18 21:20:37.3599863 +0000 UTC m=+874.967045859"
Sep 18 21:20:37.729905 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:37.729816    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:39.082111 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:39.082082    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:40.158825 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:40.158740    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:41.260800 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:41.260771    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:20:43.308638 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:43.308607    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:20:48.824561 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:20:48.824515    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:02.857415 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:02.857343    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:21:02.859404 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:02.859382    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:21:02.861208 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:02.861187    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:21:02.862966 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:02.862946    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:21:03.488511 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:03.488478    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:04.649558 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:04.649527    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:05.391571 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:05.391517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:06.355571 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:06.355526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:07.806665 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:07.806624    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:09.250272 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:09.250225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:10.535140 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:10.535108    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:12.052897 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:12.052855    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:13.136513 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:13.136483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:13.551268 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:13.551220    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:14.144882 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:14.144843    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:15.138093 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:15.138063    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:16.255300 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:16.255267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:18.358358 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:18.358309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:19.099966 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.099932    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"]
Sep 18 21:21:19.100159 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.100136    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx" podUID="790f6cb7-41ea-40f7-b955-b3e78ee4a37e" containerName="maas-api" containerID="cri-o://e74ee334bfcf26ff07c9433ec240d5d9e6f7f22ff3eb066db46d685f7bac742f" gracePeriod=30
Sep 18 21:21:19.154786 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.154759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:19.243208 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.243178    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:19.342366 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.342337    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:21:19.400130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.400050    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:19.493288 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.493256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:19.513394 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.513366    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6mfb\" (UniqueName: \"kubernetes.io/projected/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-kube-api-access-f6mfb\") pod \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") "
Sep 18 21:21:19.513581 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.513466    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-maas-api-tls\") pod \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") "
Sep 18 21:21:19.513581 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.513494    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-openshift-service-ca\") pod \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") "
Sep 18 21:21:19.513747 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.513590    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-metrics-certs\") pod \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\" (UID: \"790f6cb7-41ea-40f7-b955-b3e78ee4a37e\") "
Sep 18 21:21:19.513994 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.513967    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "790f6cb7-41ea-40f7-b955-b3e78ee4a37e" (UID: "790f6cb7-41ea-40f7-b955-b3e78ee4a37e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:21:19.515548 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.515527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-kube-api-access-f6mfb" (OuterVolumeSpecName: "kube-api-access-f6mfb") pod "790f6cb7-41ea-40f7-b955-b3e78ee4a37e" (UID: "790f6cb7-41ea-40f7-b955-b3e78ee4a37e"). InnerVolumeSpecName "kube-api-access-f6mfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:21:19.515950 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.515935    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "790f6cb7-41ea-40f7-b955-b3e78ee4a37e" (UID: "790f6cb7-41ea-40f7-b955-b3e78ee4a37e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:21:19.516011 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.515964    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "790f6cb7-41ea-40f7-b955-b3e78ee4a37e" (UID: "790f6cb7-41ea-40f7-b955-b3e78ee4a37e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:21:19.528135 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.528106    2579 generic.go:356] "Generic (PLEG): container finished" podID="790f6cb7-41ea-40f7-b955-b3e78ee4a37e" containerID="e74ee334bfcf26ff07c9433ec240d5d9e6f7f22ff3eb066db46d685f7bac742f" exitCode=0
Sep 18 21:21:19.528230 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.528154    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"
Sep 18 21:21:19.528280 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.528156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx" event={"ID":"790f6cb7-41ea-40f7-b955-b3e78ee4a37e","Type":"ContainerDied","Data":"e74ee334bfcf26ff07c9433ec240d5d9e6f7f22ff3eb066db46d685f7bac742f"}
Sep 18 21:21:19.528331 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.528281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx" event={"ID":"790f6cb7-41ea-40f7-b955-b3e78ee4a37e","Type":"ContainerDied","Data":"834e11db548a509469611061671124b0ba5866c0bae9e7f6c288710c0daf84f3"}
Sep 18 21:21:19.528331 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.528309    2579 scope.go:117] "RemoveContainer" containerID="e74ee334bfcf26ff07c9433ec240d5d9e6f7f22ff3eb066db46d685f7bac742f"
Sep 18 21:21:19.539820 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.539775    2579 scope.go:117] "RemoveContainer" containerID="e74ee334bfcf26ff07c9433ec240d5d9e6f7f22ff3eb066db46d685f7bac742f"
Sep 18 21:21:19.540540 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:21:19.540519    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74ee334bfcf26ff07c9433ec240d5d9e6f7f22ff3eb066db46d685f7bac742f\": container with ID starting with e74ee334bfcf26ff07c9433ec240d5d9e6f7f22ff3eb066db46d685f7bac742f not found: ID does not exist" containerID="e74ee334bfcf26ff07c9433ec240d5d9e6f7f22ff3eb066db46d685f7bac742f"
Sep 18 21:21:19.540607 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.540549    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74ee334bfcf26ff07c9433ec240d5d9e6f7f22ff3eb066db46d685f7bac742f"} err="failed to get container status \"e74ee334bfcf26ff07c9433ec240d5d9e6f7f22ff3eb066db46d685f7bac742f\": rpc error: code = NotFound desc = could not find container \"e74ee334bfcf26ff07c9433ec240d5d9e6f7f22ff3eb066db46d685f7bac742f\": container with ID starting with e74ee334bfcf26ff07c9433ec240d5d9e6f7f22ff3eb066db46d685f7bac742f not found: ID does not exist"
Sep 18 21:21:19.561527 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.561499    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"]
Sep 18 21:21:19.578718 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.578692    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7e01a72f-7dddc68cbc-296sx"]
Sep 18 21:21:19.615525 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.615503    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:21:19.615525 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.615525    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:21:19.615679 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.615535    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:21:19.615679 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.615544    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6mfb\" (UniqueName: \"kubernetes.io/projected/790f6cb7-41ea-40f7-b955-b3e78ee4a37e-kube-api-access-f6mfb\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:21:19.629769 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.629746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:19.942782 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:19.942752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:20.068271 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:20.068234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:20.547967 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:20.547935    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:20.821910 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:20.821828    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:20.921981 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:20.921947    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="790f6cb7-41ea-40f7-b955-b3e78ee4a37e" path="/var/lib/kubelet/pods/790f6cb7-41ea-40f7-b955-b3e78ee4a37e/volumes"
Sep 18 21:21:21.272758 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:21.272669    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:22.197671 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:22.197625    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:22.959877 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:22.959848    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:23.296150 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:23.296121    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:23.505924 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:23.505889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:24.509683 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:24.509634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:25.742211 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:25.742171    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:26.753052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.753014    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"]
Sep 18 21:21:26.754075 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.754052    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790f6cb7-41ea-40f7-b955-b3e78ee4a37e" containerName="maas-api"
Sep 18 21:21:26.754075 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.754076    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="790f6cb7-41ea-40f7-b955-b3e78ee4a37e" containerName="maas-api"
Sep 18 21:21:26.754285 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.754271    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="790f6cb7-41ea-40f7-b955-b3e78ee4a37e" containerName="maas-api"
Sep 18 21:21:26.758438 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.758414    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:26.761127 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.761102    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1084f187-serving-cert\""
Sep 18 21:21:26.766792 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.766766    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"]
Sep 18 21:21:26.782151 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.782120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c88efd0-8fb7-4c8f-8085-153052a3aa91-maas-api-tls\") pod \"maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:26.782266 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.782180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c88efd0-8fb7-4c8f-8085-153052a3aa91-metrics-certs\") pod \"maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:26.782337 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.782298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c88efd0-8fb7-4c8f-8085-153052a3aa91-openshift-service-ca\") pod \"maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:26.782396 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.782338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf26p\" (UniqueName: \"kubernetes.io/projected/2c88efd0-8fb7-4c8f-8085-153052a3aa91-kube-api-access-tf26p\") pod \"maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:26.785202 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.785177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:26.867820 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.867793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:26.883031 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.883006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c88efd0-8fb7-4c8f-8085-153052a3aa91-openshift-service-ca\") pod \"maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:26.883130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.883037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf26p\" (UniqueName: \"kubernetes.io/projected/2c88efd0-8fb7-4c8f-8085-153052a3aa91-kube-api-access-tf26p\") pod \"maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:26.883130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.883089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c88efd0-8fb7-4c8f-8085-153052a3aa91-maas-api-tls\") pod \"maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:26.883130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.883123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c88efd0-8fb7-4c8f-8085-153052a3aa91-metrics-certs\") pod \"maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:26.883625 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.883604    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c88efd0-8fb7-4c8f-8085-153052a3aa91-openshift-service-ca\") pod \"maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:26.885489 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.885468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c88efd0-8fb7-4c8f-8085-153052a3aa91-maas-api-tls\") pod \"maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:26.885489 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.885486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c88efd0-8fb7-4c8f-8085-153052a3aa91-metrics-certs\") pod \"maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:26.894558 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.894539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf26p\" (UniqueName: \"kubernetes.io/projected/2c88efd0-8fb7-4c8f-8085-153052a3aa91-kube-api-access-tf26p\") pod \"maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:26.981269 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:26.981243    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:27.070263 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:27.070236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:27.361205 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:27.361182    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"]
Sep 18 21:21:27.363029 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:21:27.363000    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c88efd0_8fb7_4c8f_8085_153052a3aa91.slice/crio-1a43976d3c020bcad763efeb322f413f60970d094568536c6758af0d94682b61 WatchSource:0}: Error finding container 1a43976d3c020bcad763efeb322f413f60970d094568536c6758af0d94682b61: Status 404 returned error can't find the container with id 1a43976d3c020bcad763efeb322f413f60970d094568536c6758af0d94682b61
Sep 18 21:21:27.369172 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:27.369146    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:27.462521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:27.462483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:27.571749 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:27.571715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4" event={"ID":"2c88efd0-8fb7-4c8f-8085-153052a3aa91","Type":"ContainerStarted","Data":"1a43976d3c020bcad763efeb322f413f60970d094568536c6758af0d94682b61"}
Sep 18 21:21:27.768541 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:27.768462    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:28.541238 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:28.541198    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:28.578816 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:28.578787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4" event={"ID":"2c88efd0-8fb7-4c8f-8085-153052a3aa91","Type":"ContainerStarted","Data":"b7912cb10b28d6a72d4e9cbcad8fd8dda18a7160f91e48922daad0499cca7524"}
Sep 18 21:21:28.578949 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:28.578908    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:28.602907 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:28.602866    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4" podStartSLOduration=1.916441071 podStartE2EDuration="2.602853765s" totalImagesPullingTime="686.412694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:21:28.601583862 +0000 UTC m=+926.208643420" watchObservedRunningTime="2026-09-18 21:21:28.602853765 +0000 UTC m=+926.209913322"
Sep 18 21:21:29.452537 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:29.452495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:34.589819 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:34.589789    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:21:36.122915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.122877    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"]
Sep 18 21:21:36.125473 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.125456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.131398 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.131371    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-21df59\"/\"kube-root-ca.crt\""
Sep 18 21:21:36.132564 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.132536    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-21df59\"/\"openshift-service-ca.crt\""
Sep 18 21:21:36.132709 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.132541    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-21df59\"/\"default-dockercfg-hztbz\""
Sep 18 21:21:36.132709 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.132577    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-21df59\"/\"autores-model-21df59-kserve-self-signed-certs\""
Sep 18 21:21:36.155123 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.155095    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"]
Sep 18 21:21:36.170510 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.170481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-dshm\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.170713 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.170524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-tls-certs\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.170713 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.170591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-home\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.170713 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.170681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-tmp-dir\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.170847 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.170725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqlps\" (UniqueName: \"kubernetes.io/projected/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-kube-api-access-bqlps\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.170887 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.170839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-model-cache\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.272144 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.272107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-home\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.272338 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.272157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-tmp-dir\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.272338 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.272184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqlps\" (UniqueName: \"kubernetes.io/projected/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-kube-api-access-bqlps\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.272338 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.272244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-model-cache\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.272338 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.272286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-dshm\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.272338 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.272304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-tls-certs\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.272610 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.272565    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-tmp-dir\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.272705 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.272617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-home\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.272705 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.272634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-model-cache\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.274530 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.274512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-dshm\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.274937 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.274913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-tls-certs\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.282160 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.282140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqlps\" (UniqueName: \"kubernetes.io/projected/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-kube-api-access-bqlps\") pod \"autores-model-21df59-kserve-9867f677f-pfbqr\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") " pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.435171 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.435075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:36.537351 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.537322    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:36.731102 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:36.731075    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"]
Sep 18 21:21:36.733771 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:21:36.733745    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d7c61bd_07bd_4d90_aac8_a8de01ad0df5.slice/crio-c056ebb34ee4234c3fb88b4a3a072f7adba9b42700c6dcf22be3283592a8a818 WatchSource:0}: Error finding container c056ebb34ee4234c3fb88b4a3a072f7adba9b42700c6dcf22be3283592a8a818: Status 404 returned error can't find the container with id c056ebb34ee4234c3fb88b4a3a072f7adba9b42700c6dcf22be3283592a8a818
Sep 18 21:21:37.259147 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:37.259113    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:37.632276 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:37.632239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr" event={"ID":"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5","Type":"ContainerStarted","Data":"2090b0adc42815b9ee2eae2e668367513aa6786b8fbff9af09025e494dfd4c06"}
Sep 18 21:21:37.632436 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:37.632285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr" event={"ID":"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5","Type":"ContainerStarted","Data":"c056ebb34ee4234c3fb88b4a3a072f7adba9b42700c6dcf22be3283592a8a818"}
Sep 18 21:21:37.677803 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:37.677755    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr" podStartSLOduration=1.677741037 podStartE2EDuration="1.677741037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:21:37.673625521 +0000 UTC m=+935.280685093" watchObservedRunningTime="2026-09-18 21:21:37.677741037 +0000 UTC m=+935.284800595"
Sep 18 21:21:46.435979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:46.435929    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:46.435979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:46.435985    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:46.449028 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:46.449000    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:46.691979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:46.691893    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:21:48.081197 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:48.081150    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:50.435264 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:50.435227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:52.712660 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:52.712618    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:53.495298 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:53.495264    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:54.721497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:54.721455    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:55.593998 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:55.593958    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:56.967246 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:56.967211    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:57.784245 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:57.784205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:58.742093 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:58.742058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:21:59.995487 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:21:59.995442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:01.288107 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:01.288071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:04.450398 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:04.450359    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:05.127541 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.127505    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"]
Sep 18 21:22:05.127815 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.127785    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4" podUID="2c88efd0-8fb7-4c8f-8085-153052a3aa91" containerName="maas-api" containerID="cri-o://b7912cb10b28d6a72d4e9cbcad8fd8dda18a7160f91e48922daad0499cca7524" gracePeriod=30
Sep 18 21:22:05.163717 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.163668    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:05.258315 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.255164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:05.349046 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.349012    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:05.381599 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.381543    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:22:05.467112 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.467065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:05.565393 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.565359    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c88efd0-8fb7-4c8f-8085-153052a3aa91-metrics-certs\") pod \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") "
Sep 18 21:22:05.565557 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.565459    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf26p\" (UniqueName: \"kubernetes.io/projected/2c88efd0-8fb7-4c8f-8085-153052a3aa91-kube-api-access-tf26p\") pod \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") "
Sep 18 21:22:05.565557 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.565488    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c88efd0-8fb7-4c8f-8085-153052a3aa91-maas-api-tls\") pod \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") "
Sep 18 21:22:05.565557 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.565549    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c88efd0-8fb7-4c8f-8085-153052a3aa91-openshift-service-ca\") pod \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\" (UID: \"2c88efd0-8fb7-4c8f-8085-153052a3aa91\") "
Sep 18 21:22:05.565979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.565936    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c88efd0-8fb7-4c8f-8085-153052a3aa91-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c88efd0-8fb7-4c8f-8085-153052a3aa91" (UID: "2c88efd0-8fb7-4c8f-8085-153052a3aa91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:22:05.567683 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.567632    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c88efd0-8fb7-4c8f-8085-153052a3aa91-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c88efd0-8fb7-4c8f-8085-153052a3aa91" (UID: "2c88efd0-8fb7-4c8f-8085-153052a3aa91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:22:05.567683 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.567660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c88efd0-8fb7-4c8f-8085-153052a3aa91-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2c88efd0-8fb7-4c8f-8085-153052a3aa91" (UID: "2c88efd0-8fb7-4c8f-8085-153052a3aa91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:22:05.567821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.567693    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c88efd0-8fb7-4c8f-8085-153052a3aa91-kube-api-access-tf26p" (OuterVolumeSpecName: "kube-api-access-tf26p") pod "2c88efd0-8fb7-4c8f-8085-153052a3aa91" (UID: "2c88efd0-8fb7-4c8f-8085-153052a3aa91"). InnerVolumeSpecName "kube-api-access-tf26p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:22:05.667431 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.667344    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf26p\" (UniqueName: \"kubernetes.io/projected/2c88efd0-8fb7-4c8f-8085-153052a3aa91-kube-api-access-tf26p\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:05.667431 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.667374    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c88efd0-8fb7-4c8f-8085-153052a3aa91-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:05.667431 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.667386    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c88efd0-8fb7-4c8f-8085-153052a3aa91-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:05.667431 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.667395    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c88efd0-8fb7-4c8f-8085-153052a3aa91-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:05.783494 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.783454    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c88efd0-8fb7-4c8f-8085-153052a3aa91" containerID="b7912cb10b28d6a72d4e9cbcad8fd8dda18a7160f91e48922daad0499cca7524" exitCode=0
Sep 18 21:22:05.783701 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.783505    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"
Sep 18 21:22:05.783701 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.783539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4" event={"ID":"2c88efd0-8fb7-4c8f-8085-153052a3aa91","Type":"ContainerDied","Data":"b7912cb10b28d6a72d4e9cbcad8fd8dda18a7160f91e48922daad0499cca7524"}
Sep 18 21:22:05.783701 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.783618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4" event={"ID":"2c88efd0-8fb7-4c8f-8085-153052a3aa91","Type":"ContainerDied","Data":"1a43976d3c020bcad763efeb322f413f60970d094568536c6758af0d94682b61"}
Sep 18 21:22:05.783701 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.783662    2579 scope.go:117] "RemoveContainer" containerID="b7912cb10b28d6a72d4e9cbcad8fd8dda18a7160f91e48922daad0499cca7524"
Sep 18 21:22:05.793419 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.793396    2579 scope.go:117] "RemoveContainer" containerID="b7912cb10b28d6a72d4e9cbcad8fd8dda18a7160f91e48922daad0499cca7524"
Sep 18 21:22:05.793705 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:22:05.793685    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7912cb10b28d6a72d4e9cbcad8fd8dda18a7160f91e48922daad0499cca7524\": container with ID starting with b7912cb10b28d6a72d4e9cbcad8fd8dda18a7160f91e48922daad0499cca7524 not found: ID does not exist" containerID="b7912cb10b28d6a72d4e9cbcad8fd8dda18a7160f91e48922daad0499cca7524"
Sep 18 21:22:05.793764 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.793715    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7912cb10b28d6a72d4e9cbcad8fd8dda18a7160f91e48922daad0499cca7524"} err="failed to get container status \"b7912cb10b28d6a72d4e9cbcad8fd8dda18a7160f91e48922daad0499cca7524\": rpc error: code = NotFound desc = could not find container \"b7912cb10b28d6a72d4e9cbcad8fd8dda18a7160f91e48922daad0499cca7524\": container with ID starting with b7912cb10b28d6a72d4e9cbcad8fd8dda18a7160f91e48922daad0499cca7524 not found: ID does not exist"
Sep 18 21:22:05.834973 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.834944    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"]
Sep 18 21:22:05.838957 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.838926    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1084f187-7b5b6d4c6b-xxjq4"]
Sep 18 21:22:05.924734 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:05.924640    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:06.923164 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:06.923126    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c88efd0-8fb7-4c8f-8085-153052a3aa91" path="/var/lib/kubelet/pods/2c88efd0-8fb7-4c8f-8085-153052a3aa91/volumes"
Sep 18 21:22:06.992512 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:06.992478    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:07.302024 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:07.301982    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:07.563439 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:07.563353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:07.755573 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:07.755541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:07.872423 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:07.872330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:08.936856 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:08.936811    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:10.455789 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:10.455752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:10.791062 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:10.791034    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:10.955975 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:10.955944    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:12.131632 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:12.131598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:13.137297 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:13.137262    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:13.884859 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:13.884823    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:14.915227 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:14.915195    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:15.889957 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:15.889918    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"]
Sep 18 21:22:15.890618 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:15.890596    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c88efd0-8fb7-4c8f-8085-153052a3aa91" containerName="maas-api"
Sep 18 21:22:15.890618 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:15.890617    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c88efd0-8fb7-4c8f-8085-153052a3aa91" containerName="maas-api"
Sep 18 21:22:15.890839 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:15.890729    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c88efd0-8fb7-4c8f-8085-153052a3aa91" containerName="maas-api"
Sep 18 21:22:15.895197 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:15.895170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:15.898506 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:15.898478    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-962c4157-serving-cert\""
Sep 18 21:22:15.914692 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:15.914665    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"]
Sep 18 21:22:15.953188 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:15.953157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:15.971150 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:15.971120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4862\" (UniqueName: \"kubernetes.io/projected/872802d7-7f66-43cd-98b5-81027f7f3c19-kube-api-access-b4862\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:15.971291 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:15.971159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-maas-api-tls\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:15.971291 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:15.971178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-metrics-certs\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:15.971367 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:15.971298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/872802d7-7f66-43cd-98b5-81027f7f3c19-openshift-service-ca\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:16.067143 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:16.067113    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:16.072314 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:16.072289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/872802d7-7f66-43cd-98b5-81027f7f3c19-openshift-service-ca\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:16.072442 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:16.072428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4862\" (UniqueName: \"kubernetes.io/projected/872802d7-7f66-43cd-98b5-81027f7f3c19-kube-api-access-b4862\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:16.072484 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:16.072461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-maas-api-tls\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:16.072484 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:16.072478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-metrics-certs\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:16.072597 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:22:16.072580    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-serving-cert: secret "maas-api-e2e-derive-962c4157-serving-cert" not found
Sep 18 21:22:16.072677 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:22:16.072665    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-maas-api-tls podName:872802d7-7f66-43cd-98b5-81027f7f3c19 nodeName:}" failed. No retries permitted until 2026-09-18 21:22:16.572635347 +0000 UTC m=+974.179694883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-maas-api-tls") pod "maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz" (UID: "872802d7-7f66-43cd-98b5-81027f7f3c19") : secret "maas-api-e2e-derive-962c4157-serving-cert" not found
Sep 18 21:22:16.073063 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:16.073043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/872802d7-7f66-43cd-98b5-81027f7f3c19-openshift-service-ca\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:16.075002 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:16.074985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-metrics-certs\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:16.081788 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:16.081766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4862\" (UniqueName: \"kubernetes.io/projected/872802d7-7f66-43cd-98b5-81027f7f3c19-kube-api-access-b4862\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:16.579063 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:16.579027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-maas-api-tls\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:16.579251 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:22:16.579175    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-serving-cert: secret "maas-api-e2e-derive-962c4157-serving-cert" not found
Sep 18 21:22:16.579293 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:22:16.579254    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-maas-api-tls podName:872802d7-7f66-43cd-98b5-81027f7f3c19 nodeName:}" failed. No retries permitted until 2026-09-18 21:22:17.579239154 +0000 UTC m=+975.186298695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-maas-api-tls") pod "maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz" (UID: "872802d7-7f66-43cd-98b5-81027f7f3c19") : secret "maas-api-e2e-derive-962c4157-serving-cert" not found
Sep 18 21:22:16.753190 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:16.753150    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:16.852185 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:16.852068    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:17.114226 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:17.114126    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:17.392994 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:17.392896    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:17.592667 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:17.592601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-maas-api-tls\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:17.595206 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:17.595178    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-maas-api-tls\") pod \"maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:17.706759 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:17.706674    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:17.785529 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:17.785492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:18.012461 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:18.012360    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"]
Sep 18 21:22:18.016725 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:22:18.016698    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872802d7_7f66_43cd_98b5_81027f7f3c19.slice/crio-19d7efd44a3038515403f451d6b9f8276c6e01cf7719e2f63fac9567ad13683a WatchSource:0}: Error finding container 19d7efd44a3038515403f451d6b9f8276c6e01cf7719e2f63fac9567ad13683a: Status 404 returned error can't find the container with id 19d7efd44a3038515403f451d6b9f8276c6e01cf7719e2f63fac9567ad13683a
Sep 18 21:22:18.018576 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:18.018552    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:22:18.853325 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:18.853291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz" event={"ID":"872802d7-7f66-43cd-98b5-81027f7f3c19","Type":"ContainerStarted","Data":"d4e73e4051e7f47b8066952744b2662e3b0f81b29b79112998cacef53cb5b69c"}
Sep 18 21:22:18.853736 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:18.853346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz" event={"ID":"872802d7-7f66-43cd-98b5-81027f7f3c19","Type":"ContainerStarted","Data":"19d7efd44a3038515403f451d6b9f8276c6e01cf7719e2f63fac9567ad13683a"}
Sep 18 21:22:18.853736 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:18.853378    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:18.855791 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:18.855767    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:19.609104 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:19.609072    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:20.459508 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.459446    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz" podStartSLOduration=4.974048484 podStartE2EDuration="5.459424925s" totalImagesPullingTime="485.376441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:22:18.87800745 +0000 UTC m=+976.485067007" watchObservedRunningTime="2026-09-18 21:22:20.459424925 +0000 UTC m=+978.066484482"
Sep 18 21:22:20.461740 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.461272    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"]
Sep 18 21:22:20.461740 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.461500    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l" podUID="2c8533bd-22ba-4af4-853c-37c5d34feb9d" containerName="maas-api" containerID="cri-o://5a036307e1ad64a80c38023f5082986ed2820b0346fde01e119381f9d4418547" gracePeriod=30
Sep 18 21:22:20.508366 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.508316    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:20.559063 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.559028    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"]
Sep 18 21:22:20.559268 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.559242    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879" podUID="6bf6eceb-c370-46a5-9726-7677e951ce5d" containerName="payload-pre-processing" containerID="cri-o://8d06bcd8349b9bf1c4502a34c4274003a369b46a4fe78a3fa634e4466102c470" gracePeriod=30
Sep 18 21:22:20.614479 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.614440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:20.738577 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.738500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:20.741369 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.741341    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:22:20.825706 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.825680    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c8533bd-22ba-4af4-853c-37c5d34feb9d-metrics-certs\") pod \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") "
Sep 18 21:22:20.825853 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.825731    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c8533bd-22ba-4af4-853c-37c5d34feb9d-maas-api-tls\") pod \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") "
Sep 18 21:22:20.825914 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.825869    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkb8l\" (UniqueName: \"kubernetes.io/projected/2c8533bd-22ba-4af4-853c-37c5d34feb9d-kube-api-access-gkb8l\") pod \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") "
Sep 18 21:22:20.825914 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.825899    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8533bd-22ba-4af4-853c-37c5d34feb9d-openshift-service-ca\") pod \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\" (UID: \"2c8533bd-22ba-4af4-853c-37c5d34feb9d\") "
Sep 18 21:22:20.826391 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.826349    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8533bd-22ba-4af4-853c-37c5d34feb9d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c8533bd-22ba-4af4-853c-37c5d34feb9d" (UID: "2c8533bd-22ba-4af4-853c-37c5d34feb9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:22:20.828259 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.828232    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8533bd-22ba-4af4-853c-37c5d34feb9d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c8533bd-22ba-4af4-853c-37c5d34feb9d" (UID: "2c8533bd-22ba-4af4-853c-37c5d34feb9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:22:20.828553 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.828535    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8533bd-22ba-4af4-853c-37c5d34feb9d-kube-api-access-gkb8l" (OuterVolumeSpecName: "kube-api-access-gkb8l") pod "2c8533bd-22ba-4af4-853c-37c5d34feb9d" (UID: "2c8533bd-22ba-4af4-853c-37c5d34feb9d"). InnerVolumeSpecName "kube-api-access-gkb8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:22:20.829811 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.829778    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8533bd-22ba-4af4-853c-37c5d34feb9d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2c8533bd-22ba-4af4-853c-37c5d34feb9d" (UID: "2c8533bd-22ba-4af4-853c-37c5d34feb9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:22:20.834230 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.834199    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"
Sep 18 21:22:20.865185 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.865145    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c8533bd-22ba-4af4-853c-37c5d34feb9d" containerID="5a036307e1ad64a80c38023f5082986ed2820b0346fde01e119381f9d4418547" exitCode=0
Sep 18 21:22:20.865379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.865199    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"
Sep 18 21:22:20.865379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.865230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l" event={"ID":"2c8533bd-22ba-4af4-853c-37c5d34feb9d","Type":"ContainerDied","Data":"5a036307e1ad64a80c38023f5082986ed2820b0346fde01e119381f9d4418547"}
Sep 18 21:22:20.865379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.865289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l" event={"ID":"2c8533bd-22ba-4af4-853c-37c5d34feb9d","Type":"ContainerDied","Data":"4899377a5a87e651d5bf748c430e9b9509a8b8e7f18695acdf1674e5814afe12"}
Sep 18 21:22:20.865379 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.865317    2579 scope.go:117] "RemoveContainer" containerID="5a036307e1ad64a80c38023f5082986ed2820b0346fde01e119381f9d4418547"
Sep 18 21:22:20.866681 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.866654    2579 generic.go:356] "Generic (PLEG): container finished" podID="6bf6eceb-c370-46a5-9726-7677e951ce5d" containerID="8d06bcd8349b9bf1c4502a34c4274003a369b46a4fe78a3fa634e4466102c470" exitCode=0
Sep 18 21:22:20.866802 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.866707    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"
Sep 18 21:22:20.866802 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.866769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879" event={"ID":"6bf6eceb-c370-46a5-9726-7677e951ce5d","Type":"ContainerDied","Data":"8d06bcd8349b9bf1c4502a34c4274003a369b46a4fe78a3fa634e4466102c470"}
Sep 18 21:22:20.866908 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.866815    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879" event={"ID":"6bf6eceb-c370-46a5-9726-7677e951ce5d","Type":"ContainerDied","Data":"d623000685264724cd7329f0cb15b7440cc174ab021209260285d8ff63011b66"}
Sep 18 21:22:20.872297 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.872274    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:20.875487 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.875471    2579 scope.go:117] "RemoveContainer" containerID="5a036307e1ad64a80c38023f5082986ed2820b0346fde01e119381f9d4418547"
Sep 18 21:22:20.875813 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:22:20.875794    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a036307e1ad64a80c38023f5082986ed2820b0346fde01e119381f9d4418547\": container with ID starting with 5a036307e1ad64a80c38023f5082986ed2820b0346fde01e119381f9d4418547 not found: ID does not exist" containerID="5a036307e1ad64a80c38023f5082986ed2820b0346fde01e119381f9d4418547"
Sep 18 21:22:20.875879 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.875821    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a036307e1ad64a80c38023f5082986ed2820b0346fde01e119381f9d4418547"} err="failed to get container status \"5a036307e1ad64a80c38023f5082986ed2820b0346fde01e119381f9d4418547\": rpc error: code = NotFound desc = could not find container \"5a036307e1ad64a80c38023f5082986ed2820b0346fde01e119381f9d4418547\": container with ID starting with 5a036307e1ad64a80c38023f5082986ed2820b0346fde01e119381f9d4418547 not found: ID does not exist"
Sep 18 21:22:20.875879 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.875846    2579 scope.go:117] "RemoveContainer" containerID="8d06bcd8349b9bf1c4502a34c4274003a369b46a4fe78a3fa634e4466102c470"
Sep 18 21:22:20.885635 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.885618    2579 scope.go:117] "RemoveContainer" containerID="8d06bcd8349b9bf1c4502a34c4274003a369b46a4fe78a3fa634e4466102c470"
Sep 18 21:22:20.886013 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:22:20.885994    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d06bcd8349b9bf1c4502a34c4274003a369b46a4fe78a3fa634e4466102c470\": container with ID starting with 8d06bcd8349b9bf1c4502a34c4274003a369b46a4fe78a3fa634e4466102c470 not found: ID does not exist" containerID="8d06bcd8349b9bf1c4502a34c4274003a369b46a4fe78a3fa634e4466102c470"
Sep 18 21:22:20.886076 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.886020    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d06bcd8349b9bf1c4502a34c4274003a369b46a4fe78a3fa634e4466102c470"} err="failed to get container status \"8d06bcd8349b9bf1c4502a34c4274003a369b46a4fe78a3fa634e4466102c470\": rpc error: code = NotFound desc = could not find container \"8d06bcd8349b9bf1c4502a34c4274003a369b46a4fe78a3fa634e4466102c470\": container with ID starting with 8d06bcd8349b9bf1c4502a34c4274003a369b46a4fe78a3fa634e4466102c470 not found: ID does not exist"
Sep 18 21:22:20.927087 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.927047    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6j2h\" (UniqueName: \"kubernetes.io/projected/6bf6eceb-c370-46a5-9726-7677e951ce5d-kube-api-access-b6j2h\") pod \"6bf6eceb-c370-46a5-9726-7677e951ce5d\" (UID: \"6bf6eceb-c370-46a5-9726-7677e951ce5d\") "
Sep 18 21:22:20.927312 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.927123    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf6eceb-c370-46a5-9726-7677e951ce5d-plugins-config-volume\") pod \"6bf6eceb-c370-46a5-9726-7677e951ce5d\" (UID: \"6bf6eceb-c370-46a5-9726-7677e951ce5d\") "
Sep 18 21:22:20.927617 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.927470    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkb8l\" (UniqueName: \"kubernetes.io/projected/2c8533bd-22ba-4af4-853c-37c5d34feb9d-kube-api-access-gkb8l\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:20.927617 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.927492    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8533bd-22ba-4af4-853c-37c5d34feb9d-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:20.927617 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.927506    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c8533bd-22ba-4af4-853c-37c5d34feb9d-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:20.927617 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.927521    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c8533bd-22ba-4af4-853c-37c5d34feb9d-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:20.927867 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.927609    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf6eceb-c370-46a5-9726-7677e951ce5d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6bf6eceb-c370-46a5-9726-7677e951ce5d" (UID: "6bf6eceb-c370-46a5-9726-7677e951ce5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:22:20.929564 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.929537    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf6eceb-c370-46a5-9726-7677e951ce5d-kube-api-access-b6j2h" (OuterVolumeSpecName: "kube-api-access-b6j2h") pod "6bf6eceb-c370-46a5-9726-7677e951ce5d" (UID: "6bf6eceb-c370-46a5-9726-7677e951ce5d"). InnerVolumeSpecName "kube-api-access-b6j2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:22:20.930609 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.930583    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"]
Sep 18 21:22:20.939971 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:20.939946    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-21df59-797b78bc5-f2j9l"]
Sep 18 21:22:21.028364 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:21.028267    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6j2h\" (UniqueName: \"kubernetes.io/projected/6bf6eceb-c370-46a5-9726-7677e951ce5d-kube-api-access-b6j2h\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:21.028364 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:21.028298    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bf6eceb-c370-46a5-9726-7677e951ce5d-plugins-config-volume\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:21.123439 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:21.123399    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:21.242299 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:21.242263    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"]
Sep 18 21:22:21.254122 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:21.254086    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-21df59-5cc7956cbb-vv879"]
Sep 18 21:22:21.487900 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:21.487870    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:22.137935 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:22.137900    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:22.922378 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:22.922338    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c8533bd-22ba-4af4-853c-37c5d34feb9d" path="/var/lib/kubelet/pods/2c8533bd-22ba-4af4-853c-37c5d34feb9d/volumes"
Sep 18 21:22:22.922820 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:22.922803    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bf6eceb-c370-46a5-9726-7677e951ce5d" path="/var/lib/kubelet/pods/6bf6eceb-c370-46a5-9726-7677e951ce5d/volumes"
Sep 18 21:22:23.092353 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:23.092319    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:23.830384 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:23.830347    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:24.095402 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:24.095319    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:24.295616 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:24.295585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:24.865162 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:24.865127    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:25.469101 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:25.469072    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:26.536415 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:26.536355    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"]
Sep 18 21:22:26.753017 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:26.752979    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:26.985850 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:26.985762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:27.902395 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:27.902356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:28.563824 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:28.563779    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:30.050582 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:30.050540    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:31.245535 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:31.245500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:32.253497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:32.253464    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:33.122947 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:33.122912    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:33.939512 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:33.939476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:34.848586 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:34.848550    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:35.635709 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:35.635680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:40.424759 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:40.424728    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:41.065896 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.065857    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"]
Sep 18 21:22:41.066221 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.066189    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz" podUID="872802d7-7f66-43cd-98b5-81027f7f3c19" containerName="maas-api" containerID="cri-o://d4e73e4051e7f47b8066952744b2662e3b0f81b29b79112998cacef53cb5b69c" gracePeriod=30
Sep 18 21:22:41.102938 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.102902    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:41.195063 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.195025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:41.276413 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.276373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:41.322356 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.322299    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:41.339484 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.339462    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/872802d7-7f66-43cd-98b5-81027f7f3c19-openshift-service-ca\") pod \"872802d7-7f66-43cd-98b5-81027f7f3c19\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") "
Sep 18 21:22:41.339602 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.339524    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4862\" (UniqueName: \"kubernetes.io/projected/872802d7-7f66-43cd-98b5-81027f7f3c19-kube-api-access-b4862\") pod \"872802d7-7f66-43cd-98b5-81027f7f3c19\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") "
Sep 18 21:22:41.339659 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.339625    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-maas-api-tls\") pod \"872802d7-7f66-43cd-98b5-81027f7f3c19\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") "
Sep 18 21:22:41.339792 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.339775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-metrics-certs\") pod \"872802d7-7f66-43cd-98b5-81027f7f3c19\" (UID: \"872802d7-7f66-43cd-98b5-81027f7f3c19\") "
Sep 18 21:22:41.339845 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.339824    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872802d7-7f66-43cd-98b5-81027f7f3c19-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "872802d7-7f66-43cd-98b5-81027f7f3c19" (UID: "872802d7-7f66-43cd-98b5-81027f7f3c19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:22:41.340128 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.340108    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/872802d7-7f66-43cd-98b5-81027f7f3c19-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:41.341786 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.341639    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "872802d7-7f66-43cd-98b5-81027f7f3c19" (UID: "872802d7-7f66-43cd-98b5-81027f7f3c19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:22:41.341884 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.341809    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872802d7-7f66-43cd-98b5-81027f7f3c19-kube-api-access-b4862" (OuterVolumeSpecName: "kube-api-access-b4862") pod "872802d7-7f66-43cd-98b5-81027f7f3c19" (UID: "872802d7-7f66-43cd-98b5-81027f7f3c19"). InnerVolumeSpecName "kube-api-access-b4862". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:22:41.341884 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.341851    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "872802d7-7f66-43cd-98b5-81027f7f3c19" (UID: "872802d7-7f66-43cd-98b5-81027f7f3c19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:22:41.378799 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.378760    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:41.441241 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.441202    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:41.441241 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.441235    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4862\" (UniqueName: \"kubernetes.io/projected/872802d7-7f66-43cd-98b5-81027f7f3c19-kube-api-access-b4862\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:41.441725 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.441249    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/872802d7-7f66-43cd-98b5-81027f7f3c19-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:41.472263 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.472233    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:41.556931 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.556892    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr" podUID="1d7c61bd-07bd-4d90-aac8-a8de01ad0df5" containerName="main" containerID="cri-o://2090b0adc42815b9ee2eae2e668367513aa6786b8fbff9af09025e494dfd4c06" gracePeriod=45
Sep 18 21:22:41.564554 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.564525    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:41.799005 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.798980    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:22:41.804741 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.804719    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:41.845238 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.845209    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-tls-certs\") pod \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") "
Sep 18 21:22:41.845388 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.845264    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-dshm\") pod \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") "
Sep 18 21:22:41.845388 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.845289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqlps\" (UniqueName: \"kubernetes.io/projected/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-kube-api-access-bqlps\") pod \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") "
Sep 18 21:22:41.845388 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.845339    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-tmp-dir\") pod \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") "
Sep 18 21:22:41.845499 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.845400    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-model-cache\") pod \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") "
Sep 18 21:22:41.845499 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.845460    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-home\") pod \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\" (UID: \"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5\") "
Sep 18 21:22:41.846128 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.846083    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-home" (OuterVolumeSpecName: "home") pod "1d7c61bd-07bd-4d90-aac8-a8de01ad0df5" (UID: "1d7c61bd-07bd-4d90-aac8-a8de01ad0df5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:22:41.846271 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.846132    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d7c61bd-07bd-4d90-aac8-a8de01ad0df5" (UID: "1d7c61bd-07bd-4d90-aac8-a8de01ad0df5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:22:41.846271 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.846172    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d7c61bd-07bd-4d90-aac8-a8de01ad0df5" (UID: "1d7c61bd-07bd-4d90-aac8-a8de01ad0df5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:22:41.848061 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.848030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-dshm" (OuterVolumeSpecName: "dshm") pod "1d7c61bd-07bd-4d90-aac8-a8de01ad0df5" (UID: "1d7c61bd-07bd-4d90-aac8-a8de01ad0df5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:22:41.848152 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.848121    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-kube-api-access-bqlps" (OuterVolumeSpecName: "kube-api-access-bqlps") pod "1d7c61bd-07bd-4d90-aac8-a8de01ad0df5" (UID: "1d7c61bd-07bd-4d90-aac8-a8de01ad0df5"). InnerVolumeSpecName "kube-api-access-bqlps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:22:41.848218 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.848188    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d7c61bd-07bd-4d90-aac8-a8de01ad0df5" (UID: "1d7c61bd-07bd-4d90-aac8-a8de01ad0df5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:22:41.947046 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.946967    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-dshm\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:41.947046 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.946995    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqlps\" (UniqueName: \"kubernetes.io/projected/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-kube-api-access-bqlps\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:41.947046 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.947027    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-tmp-dir\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:41.947046 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.947037    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-model-cache\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:41.947046 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.947044    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-home\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:41.947046 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.947052    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5-tls-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:22:41.982620 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.982585    2579 generic.go:356] "Generic (PLEG): container finished" podID="872802d7-7f66-43cd-98b5-81027f7f3c19" containerID="d4e73e4051e7f47b8066952744b2662e3b0f81b29b79112998cacef53cb5b69c" exitCode=0
Sep 18 21:22:41.982804 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.982653    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"
Sep 18 21:22:41.982804 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.982678    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz" event={"ID":"872802d7-7f66-43cd-98b5-81027f7f3c19","Type":"ContainerDied","Data":"d4e73e4051e7f47b8066952744b2662e3b0f81b29b79112998cacef53cb5b69c"}
Sep 18 21:22:41.982804 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.982727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz" event={"ID":"872802d7-7f66-43cd-98b5-81027f7f3c19","Type":"ContainerDied","Data":"19d7efd44a3038515403f451d6b9f8276c6e01cf7719e2f63fac9567ad13683a"}
Sep 18 21:22:41.982804 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.982743    2579 scope.go:117] "RemoveContainer" containerID="d4e73e4051e7f47b8066952744b2662e3b0f81b29b79112998cacef53cb5b69c"
Sep 18 21:22:41.984116 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.984097    2579 generic.go:356] "Generic (PLEG): container finished" podID="1d7c61bd-07bd-4d90-aac8-a8de01ad0df5" containerID="2090b0adc42815b9ee2eae2e668367513aa6786b8fbff9af09025e494dfd4c06" exitCode=0
Sep 18 21:22:41.984208 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.984134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr" event={"ID":"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5","Type":"ContainerDied","Data":"2090b0adc42815b9ee2eae2e668367513aa6786b8fbff9af09025e494dfd4c06"}
Sep 18 21:22:41.984208 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.984191    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr" event={"ID":"1d7c61bd-07bd-4d90-aac8-a8de01ad0df5","Type":"ContainerDied","Data":"c056ebb34ee4234c3fb88b4a3a072f7adba9b42700c6dcf22be3283592a8a818"}
Sep 18 21:22:41.984208 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.984151    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"
Sep 18 21:22:41.992089 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.992072    2579 scope.go:117] "RemoveContainer" containerID="d4e73e4051e7f47b8066952744b2662e3b0f81b29b79112998cacef53cb5b69c"
Sep 18 21:22:41.992469 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:22:41.992441    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e73e4051e7f47b8066952744b2662e3b0f81b29b79112998cacef53cb5b69c\": container with ID starting with d4e73e4051e7f47b8066952744b2662e3b0f81b29b79112998cacef53cb5b69c not found: ID does not exist" containerID="d4e73e4051e7f47b8066952744b2662e3b0f81b29b79112998cacef53cb5b69c"
Sep 18 21:22:41.992535 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.992480    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e73e4051e7f47b8066952744b2662e3b0f81b29b79112998cacef53cb5b69c"} err="failed to get container status \"d4e73e4051e7f47b8066952744b2662e3b0f81b29b79112998cacef53cb5b69c\": rpc error: code = NotFound desc = could not find container \"d4e73e4051e7f47b8066952744b2662e3b0f81b29b79112998cacef53cb5b69c\": container with ID starting with d4e73e4051e7f47b8066952744b2662e3b0f81b29b79112998cacef53cb5b69c not found: ID does not exist"
Sep 18 21:22:41.992535 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:41.992507    2579 scope.go:117] "RemoveContainer" containerID="2090b0adc42815b9ee2eae2e668367513aa6786b8fbff9af09025e494dfd4c06"
Sep 18 21:22:42.000467 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:42.000452    2579 scope.go:117] "RemoveContainer" containerID="2090b0adc42815b9ee2eae2e668367513aa6786b8fbff9af09025e494dfd4c06"
Sep 18 21:22:42.000732 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:22:42.000712    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2090b0adc42815b9ee2eae2e668367513aa6786b8fbff9af09025e494dfd4c06\": container with ID starting with 2090b0adc42815b9ee2eae2e668367513aa6786b8fbff9af09025e494dfd4c06 not found: ID does not exist" containerID="2090b0adc42815b9ee2eae2e668367513aa6786b8fbff9af09025e494dfd4c06"
Sep 18 21:22:42.000791 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:42.000738    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2090b0adc42815b9ee2eae2e668367513aa6786b8fbff9af09025e494dfd4c06"} err="failed to get container status \"2090b0adc42815b9ee2eae2e668367513aa6786b8fbff9af09025e494dfd4c06\": rpc error: code = NotFound desc = could not find container \"2090b0adc42815b9ee2eae2e668367513aa6786b8fbff9af09025e494dfd4c06\": container with ID starting with 2090b0adc42815b9ee2eae2e668367513aa6786b8fbff9af09025e494dfd4c06 not found: ID does not exist"
Sep 18 21:22:42.011198 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:42.011177    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"]
Sep 18 21:22:42.019135 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:42.019114    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-962c4157-6b98b4f65-xfqnz"]
Sep 18 21:22:42.033654 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:42.033615    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"]
Sep 18 21:22:42.037309 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:42.037288    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-21df59/autores-model-21df59-kserve-9867f677f-pfbqr"]
Sep 18 21:22:42.255521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:42.255433    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:42.354436 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:42.354382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:42.484640 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:42.484592    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:42.923446 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:42.923410    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d7c61bd-07bd-4d90-aac8-a8de01ad0df5" path="/var/lib/kubelet/pods/1d7c61bd-07bd-4d90-aac8-a8de01ad0df5/volumes"
Sep 18 21:22:42.923874 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:42.923859    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872802d7-7f66-43cd-98b5-81027f7f3c19" path="/var/lib/kubelet/pods/872802d7-7f66-43cd-98b5-81027f7f3c19/volumes"
Sep 18 21:22:42.944508 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:42.944471    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:43.885435 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:43.885401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:44.111005 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:44.110967    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:45.120247 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:45.120213    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:46.025509 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:46.025476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:50.202719 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:50.202680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:50.337109 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:50.337072    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:50.454412 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:50.454322    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:50.938910 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:50.938880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:53.418967 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:53.418929    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:22:54.177510 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:22:54.177476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:09.886877 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:09.886837    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:10.921286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:10.921241    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:11.847710 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:11.847675    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:12.691814 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:12.691779    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:13.448605 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:13.448572    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:14.346253 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:14.346216    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:14.921431 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:14.921388    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:16.946988 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:16.946952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:18.419331 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:18.419298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:18.536519 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:18.536480    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:18.720308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:18.720229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:19.173476 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.173446    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:19.445351 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.445271    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"]
Sep 18 21:23:19.445749 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.445734    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf6eceb-c370-46a5-9726-7677e951ce5d" containerName="payload-pre-processing"
Sep 18 21:23:19.445800 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.445751    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf6eceb-c370-46a5-9726-7677e951ce5d" containerName="payload-pre-processing"
Sep 18 21:23:19.445800 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.445762    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872802d7-7f66-43cd-98b5-81027f7f3c19" containerName="maas-api"
Sep 18 21:23:19.445800 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.445767    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="872802d7-7f66-43cd-98b5-81027f7f3c19" containerName="maas-api"
Sep 18 21:23:19.445800 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.445779    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8533bd-22ba-4af4-853c-37c5d34feb9d" containerName="maas-api"
Sep 18 21:23:19.445800 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.445784    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8533bd-22ba-4af4-853c-37c5d34feb9d" containerName="maas-api"
Sep 18 21:23:19.445800 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.445796    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d7c61bd-07bd-4d90-aac8-a8de01ad0df5" containerName="main"
Sep 18 21:23:19.445800 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.445802    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d7c61bd-07bd-4d90-aac8-a8de01ad0df5" containerName="main"
Sep 18 21:23:19.446033 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.445870    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8533bd-22ba-4af4-853c-37c5d34feb9d" containerName="maas-api"
Sep 18 21:23:19.446033 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.445877    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bf6eceb-c370-46a5-9726-7677e951ce5d" containerName="payload-pre-processing"
Sep 18 21:23:19.446033 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.445883    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="872802d7-7f66-43cd-98b5-81027f7f3c19" containerName="maas-api"
Sep 18 21:23:19.446033 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.445892    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d7c61bd-07bd-4d90-aac8-a8de01ad0df5" containerName="main"
Sep 18 21:23:19.449128 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.449108    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.452024 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.452006    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-773e07-openshift-default-dockercfg-kb9q2\""
Sep 18 21:23:19.465284 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.465262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"]
Sep 18 21:23:19.479127 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.479104    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:19.525203 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.525166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-workload-certs\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.525203 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.525208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7874\" (UniqueName: \"kubernetes.io/projected/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-kube-api-access-h7874\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.525355 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.525238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-data\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.525355 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.525325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-token\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.525424 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.525364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-podinfo\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.525424 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.525393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-credential-socket\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.525491 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.525447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-envoy\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.525491 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.525468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-workload-socket\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.525491 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.525486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istiod-ca-cert\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.626718 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.626677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-token\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.626908 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.626735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-podinfo\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.626908 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.626778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-credential-socket\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.626908 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.626816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-envoy\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.626908 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.626846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-workload-socket\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.626908 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.626870    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istiod-ca-cert\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.627205 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.626938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-workload-certs\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.627205 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.626966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7874\" (UniqueName: \"kubernetes.io/projected/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-kube-api-access-h7874\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.627205 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.626992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-data\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.627366 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.627255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-credential-socket\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.627432 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.627362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-workload-certs\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.627432 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.627417    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-data\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.627513 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.627459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-workload-socket\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.627692 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.627673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istiod-ca-cert\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.629154 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.629133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-envoy\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.629377 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.629360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-podinfo\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.639960 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.639940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-token\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.640187 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.640164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7874\" (UniqueName: \"kubernetes.io/projected/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-kube-api-access-h7874\") pod \"e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") " pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:19.693871 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.693834    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:19.760049 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:19.759971    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:20.072999 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.072975    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"]
Sep 18 21:23:20.075004 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:23:20.074969    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e2a5c8_d9a4_4eb2_a61c_f428eccb56ac.slice/crio-4748b58eee18a431cb01b54f3a03925d10f471bf98871d939d793d336d1525bf WatchSource:0}: Error finding container 4748b58eee18a431cb01b54f3a03925d10f471bf98871d939d793d336d1525bf: Status 404 returned error can't find the container with id 4748b58eee18a431cb01b54f3a03925d10f471bf98871d939d793d336d1525bf
Sep 18 21:23:20.077425 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.077391    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:23:20.077497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.077461    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:23:20.077497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.077490    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:23:20.193032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.192993    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" event={"ID":"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac","Type":"ContainerStarted","Data":"3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8"}
Sep 18 21:23:20.193240 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.193054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" event={"ID":"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac","Type":"ContainerStarted","Data":"4748b58eee18a431cb01b54f3a03925d10f471bf98871d939d793d336d1525bf"}
Sep 18 21:23:20.221548 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.221486    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" podStartSLOduration=1.221459934 podStartE2EDuration="1.221459934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:23:20.220160971 +0000 UTC m=+1037.827220530" watchObservedRunningTime="2026-09-18 21:23:20.221459934 +0000 UTC m=+1037.828519493"
Sep 18 21:23:20.619831 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.619789    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"]
Sep 18 21:23:20.620299 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.620092    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj" podUID="2c26ef2b-d211-402f-87ce-d635a79eb990" containerName="maas-api" containerID="cri-o://41feb33af0cc9af89dc3994b20a56bdb4703974210732917e5457f8af1a8e8cc" gracePeriod=30
Sep 18 21:23:20.663286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.663240    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:20.760496 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.760462    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:20.762762 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.762733    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:20.874690 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.874598    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:23:20.939266 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.939234    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mdpc\" (UniqueName: \"kubernetes.io/projected/2c26ef2b-d211-402f-87ce-d635a79eb990-kube-api-access-6mdpc\") pod \"2c26ef2b-d211-402f-87ce-d635a79eb990\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") "
Sep 18 21:23:20.939430 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.939283    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c26ef2b-d211-402f-87ce-d635a79eb990-openshift-service-ca\") pod \"2c26ef2b-d211-402f-87ce-d635a79eb990\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") "
Sep 18 21:23:20.939430 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.939404    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c26ef2b-d211-402f-87ce-d635a79eb990-metrics-certs\") pod \"2c26ef2b-d211-402f-87ce-d635a79eb990\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") "
Sep 18 21:23:20.939554 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.939456    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c26ef2b-d211-402f-87ce-d635a79eb990-maas-api-tls\") pod \"2c26ef2b-d211-402f-87ce-d635a79eb990\" (UID: \"2c26ef2b-d211-402f-87ce-d635a79eb990\") "
Sep 18 21:23:20.939685 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.939661    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c26ef2b-d211-402f-87ce-d635a79eb990-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c26ef2b-d211-402f-87ce-d635a79eb990" (UID: "2c26ef2b-d211-402f-87ce-d635a79eb990"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:23:20.939913 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.939892    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c26ef2b-d211-402f-87ce-d635a79eb990-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:23:20.942068 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.942028    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c26ef2b-d211-402f-87ce-d635a79eb990-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c26ef2b-d211-402f-87ce-d635a79eb990" (UID: "2c26ef2b-d211-402f-87ce-d635a79eb990"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:23:20.942068 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.942048    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c26ef2b-d211-402f-87ce-d635a79eb990-kube-api-access-6mdpc" (OuterVolumeSpecName: "kube-api-access-6mdpc") pod "2c26ef2b-d211-402f-87ce-d635a79eb990" (UID: "2c26ef2b-d211-402f-87ce-d635a79eb990"). InnerVolumeSpecName "kube-api-access-6mdpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:23:20.942247 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:20.942050    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c26ef2b-d211-402f-87ce-d635a79eb990-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2c26ef2b-d211-402f-87ce-d635a79eb990" (UID: "2c26ef2b-d211-402f-87ce-d635a79eb990"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:23:21.040982 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.040949    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c26ef2b-d211-402f-87ce-d635a79eb990-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:23:21.040982 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.040981    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mdpc\" (UniqueName: \"kubernetes.io/projected/2c26ef2b-d211-402f-87ce-d635a79eb990-kube-api-access-6mdpc\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:23:21.041183 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.040994    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c26ef2b-d211-402f-87ce-d635a79eb990-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:23:21.073235 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.073162    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:21.170190 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.170101    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:21.199003 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.198969    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c26ef2b-d211-402f-87ce-d635a79eb990" containerID="41feb33af0cc9af89dc3994b20a56bdb4703974210732917e5457f8af1a8e8cc" exitCode=0
Sep 18 21:23:21.199188 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.199030    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"
Sep 18 21:23:21.199188 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.199059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj" event={"ID":"2c26ef2b-d211-402f-87ce-d635a79eb990","Type":"ContainerDied","Data":"41feb33af0cc9af89dc3994b20a56bdb4703974210732917e5457f8af1a8e8cc"}
Sep 18 21:23:21.199188 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.199126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj" event={"ID":"2c26ef2b-d211-402f-87ce-d635a79eb990","Type":"ContainerDied","Data":"85e331cde51936f181ba3585a7969c4f5e31763a92c4c2dd35b0c13c34882e26"}
Sep 18 21:23:21.199374 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.199171    2579 scope.go:117] "RemoveContainer" containerID="41feb33af0cc9af89dc3994b20a56bdb4703974210732917e5457f8af1a8e8cc"
Sep 18 21:23:21.209215 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.209198    2579 scope.go:117] "RemoveContainer" containerID="41feb33af0cc9af89dc3994b20a56bdb4703974210732917e5457f8af1a8e8cc"
Sep 18 21:23:21.209499 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:23:21.209481    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41feb33af0cc9af89dc3994b20a56bdb4703974210732917e5457f8af1a8e8cc\": container with ID starting with 41feb33af0cc9af89dc3994b20a56bdb4703974210732917e5457f8af1a8e8cc not found: ID does not exist" containerID="41feb33af0cc9af89dc3994b20a56bdb4703974210732917e5457f8af1a8e8cc"
Sep 18 21:23:21.209542 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.209508    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41feb33af0cc9af89dc3994b20a56bdb4703974210732917e5457f8af1a8e8cc"} err="failed to get container status \"41feb33af0cc9af89dc3994b20a56bdb4703974210732917e5457f8af1a8e8cc\": rpc error: code = NotFound desc = could not find container \"41feb33af0cc9af89dc3994b20a56bdb4703974210732917e5457f8af1a8e8cc\": container with ID starting with 41feb33af0cc9af89dc3994b20a56bdb4703974210732917e5457f8af1a8e8cc not found: ID does not exist"
Sep 18 21:23:21.224975 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.224948    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"]
Sep 18 21:23:21.231108 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.231082    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7d035-dddc6f5fc-ddvwj"]
Sep 18 21:23:21.481929 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.481775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:21.583885 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.583839    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:21.766175 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:21.766098    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:22.207248 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:22.207210    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:22.208117 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:22.208094    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:22.352844 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:22.352815    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:22.445462 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:22.445416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:22.922440 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:22.922407    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c26ef2b-d211-402f-87ce-d635a79eb990" path="/var/lib/kubelet/pods/2c26ef2b-d211-402f-87ce-d635a79eb990/volumes"
Sep 18 21:23:23.551298 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:23.551265    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:24.404531 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:24.404496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:24.578158 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:24.578097    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:24.672291 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:24.672207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:25.339444 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.339386    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"]
Sep 18 21:23:25.340152 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.340124    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c26ef2b-d211-402f-87ce-d635a79eb990" containerName="maas-api"
Sep 18 21:23:25.340152 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.340148    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c26ef2b-d211-402f-87ce-d635a79eb990" containerName="maas-api"
Sep 18 21:23:25.340318 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.340236    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c26ef2b-d211-402f-87ce-d635a79eb990" containerName="maas-api"
Sep 18 21:23:25.344622 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.344600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:25.347681 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.347630    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-773e07-serving-cert\""
Sep 18 21:23:25.360377 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.360349    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"]
Sep 18 21:23:25.383398 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.383369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-maas-api-tls\") pod \"maas-api-e2e-explicit-773e07-7db486486b-wn5hx\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:25.383524 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.383410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/672246a1-47aa-40e0-88a1-123b217d4676-openshift-service-ca\") pod \"maas-api-e2e-explicit-773e07-7db486486b-wn5hx\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:25.383571 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.383552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-metrics-certs\") pod \"maas-api-e2e-explicit-773e07-7db486486b-wn5hx\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:25.383607 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.383587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4dk\" (UniqueName: \"kubernetes.io/projected/672246a1-47aa-40e0-88a1-123b217d4676-kube-api-access-8n4dk\") pod \"maas-api-e2e-explicit-773e07-7db486486b-wn5hx\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:25.404325 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.404287    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:25.484862 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.484819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-maas-api-tls\") pod \"maas-api-e2e-explicit-773e07-7db486486b-wn5hx\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:25.485316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.484868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/672246a1-47aa-40e0-88a1-123b217d4676-openshift-service-ca\") pod \"maas-api-e2e-explicit-773e07-7db486486b-wn5hx\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:25.485316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.484986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-metrics-certs\") pod \"maas-api-e2e-explicit-773e07-7db486486b-wn5hx\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:25.485316 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:23:25.485001    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-serving-cert: secret "maas-api-e2e-explicit-773e07-serving-cert" not found
Sep 18 21:23:25.485316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.485022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4dk\" (UniqueName: \"kubernetes.io/projected/672246a1-47aa-40e0-88a1-123b217d4676-kube-api-access-8n4dk\") pod \"maas-api-e2e-explicit-773e07-7db486486b-wn5hx\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:25.485316 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:23:25.485070    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-maas-api-tls podName:672246a1-47aa-40e0-88a1-123b217d4676 nodeName:}" failed. No retries permitted until 2026-09-18 21:23:25.985047986 +0000 UTC m=+1043.592107530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-maas-api-tls") pod "maas-api-e2e-explicit-773e07-7db486486b-wn5hx" (UID: "672246a1-47aa-40e0-88a1-123b217d4676") : secret "maas-api-e2e-explicit-773e07-serving-cert" not found
Sep 18 21:23:25.485613 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.485595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/672246a1-47aa-40e0-88a1-123b217d4676-openshift-service-ca\") pod \"maas-api-e2e-explicit-773e07-7db486486b-wn5hx\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:25.487378 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.487362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-metrics-certs\") pod \"maas-api-e2e-explicit-773e07-7db486486b-wn5hx\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:25.500772 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.500750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4dk\" (UniqueName: \"kubernetes.io/projected/672246a1-47aa-40e0-88a1-123b217d4676-kube-api-access-8n4dk\") pod \"maas-api-e2e-explicit-773e07-7db486486b-wn5hx\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:25.541477 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.541447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:25.756437 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.756353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:25.991424 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.991378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-maas-api-tls\") pod \"maas-api-e2e-explicit-773e07-7db486486b-wn5hx\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:25.993793 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:25.993771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-maas-api-tls\") pod \"maas-api-e2e-explicit-773e07-7db486486b-wn5hx\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:26.209227 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:26.209190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:26.255395 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:26.255364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:26.316370 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:26.316286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:26.554965 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:26.554938    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"]
Sep 18 21:23:26.557244 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:23:26.557217    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672246a1_47aa_40e0_88a1_123b217d4676.slice/crio-0ad45e9b66415434075e8c4ffc218e4378d295a54241ede76d53372dd37857de WatchSource:0}: Error finding container 0ad45e9b66415434075e8c4ffc218e4378d295a54241ede76d53372dd37857de: Status 404 returned error can't find the container with id 0ad45e9b66415434075e8c4ffc218e4378d295a54241ede76d53372dd37857de
Sep 18 21:23:27.235805 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:27.235704    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx" event={"ID":"672246a1-47aa-40e0-88a1-123b217d4676","Type":"ContainerStarted","Data":"69f48ae7faf734d667f31a4a198f0cdbab6476cf8a4abbd3d49de1ab6603c086"}
Sep 18 21:23:27.235805 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:27.235760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx" event={"ID":"672246a1-47aa-40e0-88a1-123b217d4676","Type":"ContainerStarted","Data":"0ad45e9b66415434075e8c4ffc218e4378d295a54241ede76d53372dd37857de"}
Sep 18 21:23:27.236028 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:27.235893    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:27.276508 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:27.276460    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx" podStartSLOduration=1.948929935 podStartE2EDuration="2.276442154s" totalImagesPullingTime="327.512219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:23:27.276137189 +0000 UTC m=+1044.883196751" watchObservedRunningTime="2026-09-18 21:23:27.276442154 +0000 UTC m=+1044.883501714"
Sep 18 21:23:29.224546 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:29.224512    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:29.325841 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:29.325809    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:29.597868 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:29.597832    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"]
Sep 18 21:23:29.598062 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:29.598039    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44" podUID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerName="istio-proxy" containerID="cri-o://0be8c71a11c5201bb5a870c02a9eea58ee87682ed2a84c15b3e4c500a890637b" gracePeriod=30
Sep 18 21:23:29.648069 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:29.647996    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:33.246995 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:33.246962    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:23:34.854766 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.854741    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:23:34.985456 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.985364    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgnr9\" (UniqueName: \"kubernetes.io/projected/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-kube-api-access-dgnr9\") pod \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") "
Sep 18 21:23:34.985456 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.985417    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-credential-socket\") pod \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") "
Sep 18 21:23:34.985456 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.985444    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-workload-certs\") pod \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") "
Sep 18 21:23:34.985777 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.985475    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-podinfo\") pod \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") "
Sep 18 21:23:34.985777 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.985545    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-envoy\") pod \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") "
Sep 18 21:23:34.985777 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.985578    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istiod-ca-cert\") pod \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") "
Sep 18 21:23:34.985777 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.985607    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-workload-socket\") pod \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") "
Sep 18 21:23:34.985777 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.985669    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-data\") pod \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") "
Sep 18 21:23:34.986027 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.985787    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-token\") pod \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\" (UID: \"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce\") "
Sep 18 21:23:34.986027 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.985847    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" (UID: "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:23:34.986027 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.985978    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" (UID: "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:23:34.986186 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.986082    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" (UID: "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:23:34.986287 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.986265    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-credential-socket\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:23:34.986347 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.986293    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-workload-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:23:34.986347 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.986310    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-workload-socket\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:23:34.986347 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.986292    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" (UID: "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:23:34.986347 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.986325    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" (UID: "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:23:34.987862 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.987837    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-kube-api-access-dgnr9" (OuterVolumeSpecName: "kube-api-access-dgnr9") pod "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" (UID: "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce"). InnerVolumeSpecName "kube-api-access-dgnr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:23:34.987991 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.987861    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" (UID: "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:23:34.987991 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.987934    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" (UID: "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:23:34.988167 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:34.988148    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" (UID: "6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 21:23:35.087303 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.087270    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-token\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:23:35.087303 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.087295    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgnr9\" (UniqueName: \"kubernetes.io/projected/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-kube-api-access-dgnr9\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:23:35.087303 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.087305    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-podinfo\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:23:35.087303 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.087313    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-envoy\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:23:35.087557 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.087320    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istiod-ca-cert\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:23:35.087557 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.087328    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce-istio-data\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:23:35.281240 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.281141    2579 generic.go:356] "Generic (PLEG): container finished" podID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerID="0be8c71a11c5201bb5a870c02a9eea58ee87682ed2a84c15b3e4c500a890637b" exitCode=0
Sep 18 21:23:35.281240 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.281195    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"
Sep 18 21:23:35.281483 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.281237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44" event={"ID":"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce","Type":"ContainerDied","Data":"0be8c71a11c5201bb5a870c02a9eea58ee87682ed2a84c15b3e4c500a890637b"}
Sep 18 21:23:35.281483 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.281304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44" event={"ID":"6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce","Type":"ContainerDied","Data":"c76e2b40606d0dcda375c2e6c55427fe4001e3ea3304993798b60019d1635c6a"}
Sep 18 21:23:35.281483 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.281334    2579 scope.go:117] "RemoveContainer" containerID="0be8c71a11c5201bb5a870c02a9eea58ee87682ed2a84c15b3e4c500a890637b"
Sep 18 21:23:35.292499 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.292425    2579 scope.go:117] "RemoveContainer" containerID="7840e9992736df733e47a107905755093ada63a0143c987579ffa4c7a5a57936"
Sep 18 21:23:35.300959 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.300940    2579 scope.go:117] "RemoveContainer" containerID="0be8c71a11c5201bb5a870c02a9eea58ee87682ed2a84c15b3e4c500a890637b"
Sep 18 21:23:35.301212 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:23:35.301195    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be8c71a11c5201bb5a870c02a9eea58ee87682ed2a84c15b3e4c500a890637b\": container with ID starting with 0be8c71a11c5201bb5a870c02a9eea58ee87682ed2a84c15b3e4c500a890637b not found: ID does not exist" containerID="0be8c71a11c5201bb5a870c02a9eea58ee87682ed2a84c15b3e4c500a890637b"
Sep 18 21:23:35.301247 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.301221    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be8c71a11c5201bb5a870c02a9eea58ee87682ed2a84c15b3e4c500a890637b"} err="failed to get container status \"0be8c71a11c5201bb5a870c02a9eea58ee87682ed2a84c15b3e4c500a890637b\": rpc error: code = NotFound desc = could not find container \"0be8c71a11c5201bb5a870c02a9eea58ee87682ed2a84c15b3e4c500a890637b\": container with ID starting with 0be8c71a11c5201bb5a870c02a9eea58ee87682ed2a84c15b3e4c500a890637b not found: ID does not exist"
Sep 18 21:23:35.301247 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.301237    2579 scope.go:117] "RemoveContainer" containerID="7840e9992736df733e47a107905755093ada63a0143c987579ffa4c7a5a57936"
Sep 18 21:23:35.301469 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:23:35.301452    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7840e9992736df733e47a107905755093ada63a0143c987579ffa4c7a5a57936\": container with ID starting with 7840e9992736df733e47a107905755093ada63a0143c987579ffa4c7a5a57936 not found: ID does not exist" containerID="7840e9992736df733e47a107905755093ada63a0143c987579ffa4c7a5a57936"
Sep 18 21:23:35.301506 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.301473    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7840e9992736df733e47a107905755093ada63a0143c987579ffa4c7a5a57936"} err="failed to get container status \"7840e9992736df733e47a107905755093ada63a0143c987579ffa4c7a5a57936\": rpc error: code = NotFound desc = could not find container \"7840e9992736df733e47a107905755093ada63a0143c987579ffa4c7a5a57936\": container with ID starting with 7840e9992736df733e47a107905755093ada63a0143c987579ffa4c7a5a57936 not found: ID does not exist"
Sep 18 21:23:35.322100 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.322077    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"]
Sep 18 21:23:35.327998 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:35.327975    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7d035-openshift-default-5c5ddd5fbd-f9r44"]
Sep 18 21:23:36.923798 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:36.923753    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" path="/var/lib/kubelet/pods/6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce/volumes"
Sep 18 21:23:40.497453 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:40.497421    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:45.498994 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:23:45.498963    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e2a5c8_d9a4_4eb2_a61c_f428eccb56ac.slice/crio-3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@1-18662-0.service\": RecentStats: unable to find data in memory cache]"
Sep 18 21:23:46.342501 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:46.342465    2579 generic.go:356] "Generic (PLEG): container finished" podID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerID="3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8" exitCode=0
Sep 18 21:23:46.342724 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:46.342541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" event={"ID":"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac","Type":"ContainerDied","Data":"3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8"}
Sep 18 21:23:46.343074 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:46.343054    2579 scope.go:117] "RemoveContainer" containerID="3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8"
Sep 18 21:23:46.344183 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:46.344154    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:23:46.344236 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:46.344223    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:23:46.344281 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:46.344257    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:23:47.352099 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:47.352062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" event={"ID":"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac","Type":"ContainerStarted","Data":"38b57b49e007b6a83a9e56127b8c1cc7f668cdc7db5c5ed1b7c9980b8e6ae661"}
Sep 18 21:23:47.646330 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:47.646228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:47.760355 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:47.760310    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:47.761506 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:47.761478    2579 patch_prober.go:28] interesting pod/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 18 21:23:47.761668 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:47.761526    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 18 21:23:47.779976 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:47.779939    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:47.877363 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:47.877330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:48.761085 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:48.761049    2579 patch_prober.go:28] interesting pod/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 18 21:23:48.761450 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:48.761100    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 18 21:23:49.760968 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:49.760921    2579 patch_prober.go:28] interesting pod/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 18 21:23:49.761186 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:49.761006    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 18 21:23:50.760599 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:50.760563    2579 patch_prober.go:28] interesting pod/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 18 21:23:50.760805 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:50.760622    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 18 21:23:50.802502 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:50.802465    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:51.760751 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:51.760714    2579 patch_prober.go:28] interesting pod/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 18 21:23:51.760950 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:51.760772    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 18 21:23:51.835715 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:51.835678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:52.207300 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:52.207267    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:52.412185 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:52.412150    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:52.760669 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:52.760621    2579 patch_prober.go:28] interesting pod/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 18 21:23:52.760877 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:52.760694    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 18 21:23:52.936747 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:52.936708    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:53.128557 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:53.128519    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:53.764707 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:53.764675    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:53.765501 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:53.765480    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:23:54.170429 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:54.170399    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:54.450726 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:54.450613    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:54.981333 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:54.981294    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:58.362916 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:58.362881    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:23:59.130743 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:23:59.130704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:01.383184 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:01.383152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:06.812366 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:06.812333    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:07.143054 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:07.142963    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:07.329311 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:07.329274    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:07.900693 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:07.900634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:08.109776 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:08.109740    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:09.656541 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:09.656502    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:10.597987 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:10.597953    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:10.703375 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:10.703331    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:14.495855 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.495824    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"]
Sep 18 21:24:14.496298 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.496285    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerName="istio-proxy"
Sep 18 21:24:14.496350 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.496300    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerName="istio-proxy"
Sep 18 21:24:14.496350 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.496318    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerName="istio-proxy"
Sep 18 21:24:14.496350 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.496324    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerName="istio-proxy"
Sep 18 21:24:14.496472 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.496393    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerName="istio-proxy"
Sep 18 21:24:14.496472 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.496405    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6638fb-3ff8-451d-8ab7-63ea4f2de3ce" containerName="istio-proxy"
Sep 18 21:24:14.500946 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.500916    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.504206 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.504179    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1f86a8\"/\"test-model-1f86a8-kserve-self-signed-certs\""
Sep 18 21:24:14.505296 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.505259    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1f86a8\"/\"openshift-service-ca.crt\""
Sep 18 21:24:14.505296 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.505269    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1f86a8\"/\"kube-root-ca.crt\""
Sep 18 21:24:14.505772 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.505752    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1f86a8\"/\"default-dockercfg-6wmkg\""
Sep 18 21:24:14.523164 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.523140    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"]
Sep 18 21:24:14.559028 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.558996    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:14.590995 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.590960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-dshm\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.591170 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.591007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-tmp-dir\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.591170 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.591104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-tls-certs\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.591170 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.591140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-home\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.591333 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.591227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vjn4\" (UniqueName: \"kubernetes.io/projected/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-kube-api-access-2vjn4\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.591333 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.591270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-model-cache\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.692558 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.692523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-dshm\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.692747 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.692567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-tmp-dir\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.692747 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.692638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-tls-certs\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.692747 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.692709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-home\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.692883 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.692779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjn4\" (UniqueName: \"kubernetes.io/projected/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-kube-api-access-2vjn4\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.692883 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.692823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-model-cache\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.693085 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.693057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-tmp-dir\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.693219 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.693105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-home\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.693219 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.693142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-model-cache\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.694900 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.694884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-dshm\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.695147 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.695131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-tls-certs\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.704492 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.704472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjn4\" (UniqueName: \"kubernetes.io/projected/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-kube-api-access-2vjn4\") pod \"test-model-1f86a8-kserve-77d7d47dfc-nkn8t\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") " pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:14.811793 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:14.811762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:15.084587 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:15.084515    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"]
Sep 18 21:24:15.086907 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:24:15.086874    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod333efa31_f459_40ce_9a4d_94c2a1fd7ebf.slice/crio-271eb8850eed110ae8ff00a5296a20d5bacd0803a0ccd3db52c39a841959851a WatchSource:0}: Error finding container 271eb8850eed110ae8ff00a5296a20d5bacd0803a0ccd3db52c39a841959851a: Status 404 returned error can't find the container with id 271eb8850eed110ae8ff00a5296a20d5bacd0803a0ccd3db52c39a841959851a
Sep 18 21:24:15.468235 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:15.468156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:15.502021 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:15.501980    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t" event={"ID":"333efa31-f459-40ce-9a4d-94c2a1fd7ebf","Type":"ContainerStarted","Data":"207fb1f8b97ab4b4595cdebcb29a7e1a38186839224224bf1451ae01fc7372c9"}
Sep 18 21:24:15.502401 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:15.502028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t" event={"ID":"333efa31-f459-40ce-9a4d-94c2a1fd7ebf","Type":"ContainerStarted","Data":"271eb8850eed110ae8ff00a5296a20d5bacd0803a0ccd3db52c39a841959851a"}
Sep 18 21:24:15.527887 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:15.527845    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t" podStartSLOduration=1.5278307199999999 podStartE2EDuration="1.52783072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:24:15.526072595 +0000 UTC m=+1093.133132153" watchObservedRunningTime="2026-09-18 21:24:15.52783072 +0000 UTC m=+1093.134890277"
Sep 18 21:24:15.567688 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:15.567656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:16.256853 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:16.256818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:16.356570 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:16.356541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:16.949516 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:16.949483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:22.980239 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:22.980203    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:23.061304 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:23.061241    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:23.325195 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:23.325165    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:23.598848 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:23.598751    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:23.928197 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:23.928109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:24.812198 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:24.812161    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:24.812600 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:24.812218    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:24.824745 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:24.824716    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:25.564264 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:25.564228    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:24:29.573864 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.573827    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"]
Sep 18 21:24:29.577888 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.577863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.585066 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.585044    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-773e07\"/\"kube-root-ca.crt\""
Sep 18 21:24:29.586266 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.586244    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-773e07\"/\"default-dockercfg-tqt2b\""
Sep 18 21:24:29.586376 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.586293    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-773e07\"/\"explicit-model-773e07-kserve-self-signed-certs\""
Sep 18 21:24:29.586547 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.586524    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-773e07\"/\"openshift-service-ca.crt\""
Sep 18 21:24:29.593698 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.593675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"]
Sep 18 21:24:29.607976 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.607954    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:29.647225 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.647199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-tls-certs\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.647349 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.647238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-home\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.647349 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.647260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-tmp-dir\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.647440 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.647382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-dshm\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.647440 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.647426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hq2g\" (UniqueName: \"kubernetes.io/projected/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-kube-api-access-2hq2g\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.647527 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.647460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-model-cache\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.748734 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.748694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-home\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.748734 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.748736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-tmp-dir\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.748959 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.748787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-dshm\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.748959 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.748817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hq2g\" (UniqueName: \"kubernetes.io/projected/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-kube-api-access-2hq2g\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.748959 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.748845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-model-cache\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.748959 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.748911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-tls-certs\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.749197 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.749172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-home\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.749397 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.749364    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-tmp-dir\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.749528 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.749394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-model-cache\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.751120 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.751097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-dshm\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.751358 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.751342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-tls-certs\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.756977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.756956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hq2g\" (UniqueName: \"kubernetes.io/projected/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-kube-api-access-2hq2g\") pod \"explicit-model-773e07-kserve-696544f94c-5xzbd\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") " pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:29.888450 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:29.888337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:30.169922 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:30.169894    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"]
Sep 18 21:24:30.171545 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:24:30.171515    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bf98fb9_40a7_4a01_b5c6_99e7066b6106.slice/crio-e47e566045ddf363ba2a3519537136f8203d6527f1824a023efc2d390e5b66e2 WatchSource:0}: Error finding container e47e566045ddf363ba2a3519537136f8203d6527f1824a023efc2d390e5b66e2: Status 404 returned error can't find the container with id e47e566045ddf363ba2a3519537136f8203d6527f1824a023efc2d390e5b66e2
Sep 18 21:24:30.340264 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:30.340225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:30.582150 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:30.582112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd" event={"ID":"0bf98fb9-40a7-4a01-b5c6-99e7066b6106","Type":"ContainerStarted","Data":"b0fd8707663361403e095953467ff6b8e9edb6d1ea1f11d9aaa1ca21b4cb4d6a"}
Sep 18 21:24:30.582545 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:30.582160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd" event={"ID":"0bf98fb9-40a7-4a01-b5c6-99e7066b6106","Type":"ContainerStarted","Data":"e47e566045ddf363ba2a3519537136f8203d6527f1824a023efc2d390e5b66e2"}
Sep 18 21:24:30.613135 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:30.613086    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd" podStartSLOduration=1.613071079 podStartE2EDuration="1.613071079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:24:30.611794478 +0000 UTC m=+1108.218854048" watchObservedRunningTime="2026-09-18 21:24:30.613071079 +0000 UTC m=+1108.220130677"
Sep 18 21:24:31.050885 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:31.050854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:31.298620 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:31.298584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:31.398124 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:31.398051    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:31.639194 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:31.639159    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:32.566350 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:32.566318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:34.262054 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:34.262019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:35.333550 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:35.333515    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:36.804423 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:36.804389    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"]
Sep 18 21:24:36.807979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:36.807952    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:36.840257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:36.840231    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a2b92ced-serving-cert\""
Sep 18 21:24:36.852811 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:36.852786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"]
Sep 18 21:24:36.926608 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:36.926571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a0c72d-d89f-4feb-a24c-f182fab7ae25-openshift-service-ca\") pod \"maas-api-e2e-mt-a2b92ced-64975674b-t5jt6\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:36.926799 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:36.926618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-metrics-certs\") pod \"maas-api-e2e-mt-a2b92ced-64975674b-t5jt6\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:36.926799 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:36.926684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-maas-api-tls\") pod \"maas-api-e2e-mt-a2b92ced-64975674b-t5jt6\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:36.926799 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:36.926751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bwh2\" (UniqueName: \"kubernetes.io/projected/32a0c72d-d89f-4feb-a24c-f182fab7ae25-kube-api-access-4bwh2\") pod \"maas-api-e2e-mt-a2b92ced-64975674b-t5jt6\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:36.982962 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:36.982918    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:37.027996 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.027957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a0c72d-d89f-4feb-a24c-f182fab7ae25-openshift-service-ca\") pod \"maas-api-e2e-mt-a2b92ced-64975674b-t5jt6\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:37.028168 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.028013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-metrics-certs\") pod \"maas-api-e2e-mt-a2b92ced-64975674b-t5jt6\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:37.028168 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.028069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-maas-api-tls\") pod \"maas-api-e2e-mt-a2b92ced-64975674b-t5jt6\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:37.028168 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.028132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bwh2\" (UniqueName: \"kubernetes.io/projected/32a0c72d-d89f-4feb-a24c-f182fab7ae25-kube-api-access-4bwh2\") pod \"maas-api-e2e-mt-a2b92ced-64975674b-t5jt6\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:37.028358 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:24:37.028241    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-serving-cert: secret "maas-api-e2e-mt-a2b92ced-serving-cert" not found
Sep 18 21:24:37.028358 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:24:37.028319    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-maas-api-tls podName:32a0c72d-d89f-4feb-a24c-f182fab7ae25 nodeName:}" failed. No retries permitted until 2026-09-18 21:24:37.528296083 +0000 UTC m=+1115.135355634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-maas-api-tls") pod "maas-api-e2e-mt-a2b92ced-64975674b-t5jt6" (UID: "32a0c72d-d89f-4feb-a24c-f182fab7ae25") : secret "maas-api-e2e-mt-a2b92ced-serving-cert" not found
Sep 18 21:24:37.028905 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.028874    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a0c72d-d89f-4feb-a24c-f182fab7ae25-openshift-service-ca\") pod \"maas-api-e2e-mt-a2b92ced-64975674b-t5jt6\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:37.030888 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.030853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-metrics-certs\") pod \"maas-api-e2e-mt-a2b92ced-64975674b-t5jt6\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:37.051318 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.051282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bwh2\" (UniqueName: \"kubernetes.io/projected/32a0c72d-d89f-4feb-a24c-f182fab7ae25-kube-api-access-4bwh2\") pod \"maas-api-e2e-mt-a2b92ced-64975674b-t5jt6\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:37.235391 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.235309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:37.532982 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.532893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-maas-api-tls\") pod \"maas-api-e2e-mt-a2b92ced-64975674b-t5jt6\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:37.535206 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.535175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-maas-api-tls\") pod \"maas-api-e2e-mt-a2b92ced-64975674b-t5jt6\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:37.539045 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.539021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:37.718591 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.718559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:37.803930 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.803830    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:37.895790 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:37.895758    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:38.004223 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:38.004198    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"]
Sep 18 21:24:38.006604 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:24:38.006574    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a0c72d_d89f_4feb_a24c_f182fab7ae25.slice/crio-d7a5a11a019c9ab6cbf8194065caa5026d530d891b3fd49573b0efc1f037a7bc WatchSource:0}: Error finding container d7a5a11a019c9ab6cbf8194065caa5026d530d891b3fd49573b0efc1f037a7bc: Status 404 returned error can't find the container with id d7a5a11a019c9ab6cbf8194065caa5026d530d891b3fd49573b0efc1f037a7bc
Sep 18 21:24:38.626446 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:38.626400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6" event={"ID":"32a0c72d-d89f-4feb-a24c-f182fab7ae25","Type":"ContainerStarted","Data":"310dbc8ebcdb8d20199a6540efba25d922338fade1d9342efa779ced58939096"}
Sep 18 21:24:38.626620 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:38.626469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6" event={"ID":"32a0c72d-d89f-4feb-a24c-f182fab7ae25","Type":"ContainerStarted","Data":"d7a5a11a019c9ab6cbf8194065caa5026d530d891b3fd49573b0efc1f037a7bc"}
Sep 18 21:24:38.626620 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:38.626603    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:38.653587 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:38.653493    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6" podStartSLOduration=2.2720890750000002 podStartE2EDuration="2.653477539s" totalImagesPullingTime="381.388464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:24:38.653172982 +0000 UTC m=+1116.260232539" watchObservedRunningTime="2026-09-18 21:24:38.653477539 +0000 UTC m=+1116.260537097"
Sep 18 21:24:39.888468 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:39.888436    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:39.888849 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:39.888755    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:39.901314 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:39.901290    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:40.648258 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:40.648230    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:24:44.637565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:44.637535    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:24:53.287150 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:53.287112    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:24:59.043680 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:24:59.043629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:00.308077 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:00.308037    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:01.051474 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:01.051439    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:01.233385 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:01.233337    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:01.819525 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:01.819485    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:01.924628 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:01.924596    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:02.193477 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:02.193393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:02.311631 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:02.311598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:03.127443 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:03.127410    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:03.948599 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:03.948567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:04.720506 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:04.720476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:05.711702 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:05.711668    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:06.791621 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:06.791586    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:07.598742 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:07.598705    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:08.767550 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:08.767520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:09.779811 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:09.779775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:09.918906 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:09.918858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:10.038407 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:10.038377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:10.859515 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:10.859469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:12.219936 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:12.219903    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:12.710651 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:12.710604    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"]
Sep 18 21:25:12.738391 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:12.738355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:13.059919 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:13.059889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:13.877909 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:13.877878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:15.033061 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:15.033023    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:15.757599 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:15.757564    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:16.498111 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.498068    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"]
Sep 18 21:25:16.498568 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.498334    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx" podUID="672246a1-47aa-40e0-88a1-123b217d4676" containerName="maas-api" containerID="cri-o://69f48ae7faf734d667f31a4a198f0cdbab6476cf8a4abbd3d49de1ab6603c086" gracePeriod=30
Sep 18 21:25:16.549359 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.549319    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:16.752512 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.752452    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:25:16.758402 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.758374    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:16.836231 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.836190    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/672246a1-47aa-40e0-88a1-123b217d4676-openshift-service-ca\") pod \"672246a1-47aa-40e0-88a1-123b217d4676\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") "
Sep 18 21:25:16.836480 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.836455    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n4dk\" (UniqueName: \"kubernetes.io/projected/672246a1-47aa-40e0-88a1-123b217d4676-kube-api-access-8n4dk\") pod \"672246a1-47aa-40e0-88a1-123b217d4676\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") "
Sep 18 21:25:16.836696 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.836676    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-metrics-certs\") pod \"672246a1-47aa-40e0-88a1-123b217d4676\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") "
Sep 18 21:25:16.836848 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.836734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672246a1-47aa-40e0-88a1-123b217d4676-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "672246a1-47aa-40e0-88a1-123b217d4676" (UID: "672246a1-47aa-40e0-88a1-123b217d4676"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:25:16.836916 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.836831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-maas-api-tls\") pod \"672246a1-47aa-40e0-88a1-123b217d4676\" (UID: \"672246a1-47aa-40e0-88a1-123b217d4676\") "
Sep 18 21:25:16.837281 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.837257    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/672246a1-47aa-40e0-88a1-123b217d4676-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:16.839162 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.839134    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672246a1-47aa-40e0-88a1-123b217d4676-kube-api-access-8n4dk" (OuterVolumeSpecName: "kube-api-access-8n4dk") pod "672246a1-47aa-40e0-88a1-123b217d4676" (UID: "672246a1-47aa-40e0-88a1-123b217d4676"). InnerVolumeSpecName "kube-api-access-8n4dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:25:16.839361 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.839338    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "672246a1-47aa-40e0-88a1-123b217d4676" (UID: "672246a1-47aa-40e0-88a1-123b217d4676"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:25:16.839539 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.839516    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "672246a1-47aa-40e0-88a1-123b217d4676" (UID: "672246a1-47aa-40e0-88a1-123b217d4676"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:25:16.841095 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.841068    2579 generic.go:356] "Generic (PLEG): container finished" podID="672246a1-47aa-40e0-88a1-123b217d4676" containerID="69f48ae7faf734d667f31a4a198f0cdbab6476cf8a4abbd3d49de1ab6603c086" exitCode=0
Sep 18 21:25:16.841205 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.841125    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"
Sep 18 21:25:16.841205 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.841154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx" event={"ID":"672246a1-47aa-40e0-88a1-123b217d4676","Type":"ContainerDied","Data":"69f48ae7faf734d667f31a4a198f0cdbab6476cf8a4abbd3d49de1ab6603c086"}
Sep 18 21:25:16.841327 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.841231    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx" event={"ID":"672246a1-47aa-40e0-88a1-123b217d4676","Type":"ContainerDied","Data":"0ad45e9b66415434075e8c4ffc218e4378d295a54241ede76d53372dd37857de"}
Sep 18 21:25:16.841327 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.841258    2579 scope.go:117] "RemoveContainer" containerID="69f48ae7faf734d667f31a4a198f0cdbab6476cf8a4abbd3d49de1ab6603c086"
Sep 18 21:25:16.861964 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.861944    2579 scope.go:117] "RemoveContainer" containerID="69f48ae7faf734d667f31a4a198f0cdbab6476cf8a4abbd3d49de1ab6603c086"
Sep 18 21:25:16.862252 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:25:16.862232    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f48ae7faf734d667f31a4a198f0cdbab6476cf8a4abbd3d49de1ab6603c086\": container with ID starting with 69f48ae7faf734d667f31a4a198f0cdbab6476cf8a4abbd3d49de1ab6603c086 not found: ID does not exist" containerID="69f48ae7faf734d667f31a4a198f0cdbab6476cf8a4abbd3d49de1ab6603c086"
Sep 18 21:25:16.862293 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.862260    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f48ae7faf734d667f31a4a198f0cdbab6476cf8a4abbd3d49de1ab6603c086"} err="failed to get container status \"69f48ae7faf734d667f31a4a198f0cdbab6476cf8a4abbd3d49de1ab6603c086\": rpc error: code = NotFound desc = could not find container \"69f48ae7faf734d667f31a4a198f0cdbab6476cf8a4abbd3d49de1ab6603c086\": container with ID starting with 69f48ae7faf734d667f31a4a198f0cdbab6476cf8a4abbd3d49de1ab6603c086 not found: ID does not exist"
Sep 18 21:25:16.898495 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.898462    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"]
Sep 18 21:25:16.905151 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.905127    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-773e07-7db486486b-wn5hx"]
Sep 18 21:25:16.921913 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.921892    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="672246a1-47aa-40e0-88a1-123b217d4676" path="/var/lib/kubelet/pods/672246a1-47aa-40e0-88a1-123b217d4676/volumes"
Sep 18 21:25:16.938317 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.938297    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n4dk\" (UniqueName: \"kubernetes.io/projected/672246a1-47aa-40e0-88a1-123b217d4676-kube-api-access-8n4dk\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:16.938385 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.938318    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:16.938385 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.938331    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/672246a1-47aa-40e0-88a1-123b217d4676-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:16.961824 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:16.961798    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:17.138434 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:17.138398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:17.253387 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:17.253351    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:17.530521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:17.530435    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:18.221387 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:18.221349    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:19.043537 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:19.043504    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:20.069477 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:20.069443    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:20.838678 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:20.838633    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:21.642817 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.642779    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"]
Sep 18 21:25:21.643185 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.642996    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6" podUID="32a0c72d-d89f-4feb-a24c-f182fab7ae25" containerName="maas-api" containerID="cri-o://310dbc8ebcdb8d20199a6540efba25d922338fade1d9342efa779ced58939096" gracePeriod=30
Sep 18 21:25:21.715853 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.715823    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:21.869282 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.869250    2579 generic.go:356] "Generic (PLEG): container finished" podID="32a0c72d-d89f-4feb-a24c-f182fab7ae25" containerID="310dbc8ebcdb8d20199a6540efba25d922338fade1d9342efa779ced58939096" exitCode=0
Sep 18 21:25:21.869442 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.869320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6" event={"ID":"32a0c72d-d89f-4feb-a24c-f182fab7ae25","Type":"ContainerDied","Data":"310dbc8ebcdb8d20199a6540efba25d922338fade1d9342efa779ced58939096"}
Sep 18 21:25:21.877417 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.877386    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:21.907817 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.907764    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:25:21.985185 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.985152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:21.992244 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.992223    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a0c72d-d89f-4feb-a24c-f182fab7ae25-openshift-service-ca\") pod \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") "
Sep 18 21:25:21.992320 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.992289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bwh2\" (UniqueName: \"kubernetes.io/projected/32a0c72d-d89f-4feb-a24c-f182fab7ae25-kube-api-access-4bwh2\") pod \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") "
Sep 18 21:25:21.992356 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.992326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-maas-api-tls\") pod \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") "
Sep 18 21:25:21.992356 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.992344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-metrics-certs\") pod \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\" (UID: \"32a0c72d-d89f-4feb-a24c-f182fab7ae25\") "
Sep 18 21:25:21.992654 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.992616    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a0c72d-d89f-4feb-a24c-f182fab7ae25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32a0c72d-d89f-4feb-a24c-f182fab7ae25" (UID: "32a0c72d-d89f-4feb-a24c-f182fab7ae25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:25:21.992777 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.992756    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a0c72d-d89f-4feb-a24c-f182fab7ae25-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:21.994389 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.994356    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32a0c72d-d89f-4feb-a24c-f182fab7ae25" (UID: "32a0c72d-d89f-4feb-a24c-f182fab7ae25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:25:21.994389 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.994371    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a0c72d-d89f-4feb-a24c-f182fab7ae25-kube-api-access-4bwh2" (OuterVolumeSpecName: "kube-api-access-4bwh2") pod "32a0c72d-d89f-4feb-a24c-f182fab7ae25" (UID: "32a0c72d-d89f-4feb-a24c-f182fab7ae25"). InnerVolumeSpecName "kube-api-access-4bwh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:25:21.994389 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:21.994384    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "32a0c72d-d89f-4feb-a24c-f182fab7ae25" (UID: "32a0c72d-d89f-4feb-a24c-f182fab7ae25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:25:22.093443 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:22.093409    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bwh2\" (UniqueName: \"kubernetes.io/projected/32a0c72d-d89f-4feb-a24c-f182fab7ae25-kube-api-access-4bwh2\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:22.093443 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:22.093437    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:22.093443 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:22.093446    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32a0c72d-d89f-4feb-a24c-f182fab7ae25-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:22.103622 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:22.103593    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:22.187844 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:22.187759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:22.432444 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:22.432406    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:22.540708 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:22.540678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:22.621902 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:22.621868    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:22.876127 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:22.876042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6" event={"ID":"32a0c72d-d89f-4feb-a24c-f182fab7ae25","Type":"ContainerDied","Data":"d7a5a11a019c9ab6cbf8194065caa5026d530d891b3fd49573b0efc1f037a7bc"}
Sep 18 21:25:22.876127 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:22.876126    2579 scope.go:117] "RemoveContainer" containerID="310dbc8ebcdb8d20199a6540efba25d922338fade1d9342efa779ced58939096"
Sep 18 21:25:22.876566 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:22.876053    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"
Sep 18 21:25:22.937834 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:22.937806    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"]
Sep 18 21:25:22.939787 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:22.939758    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a2b92ced-64975674b-t5jt6"]
Sep 18 21:25:23.419720 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:23.419684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:24.434814 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:24.434778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:24.922967 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:24.922933    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32a0c72d-d89f-4feb-a24c-f182fab7ae25" path="/var/lib/kubelet/pods/32a0c72d-d89f-4feb-a24c-f182fab7ae25/volumes"
Sep 18 21:25:25.669482 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:25.669448    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:25.794285 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:25.794245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:27.730119 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:27.730077    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd" podUID="0bf98fb9-40a7-4a01-b5c6-99e7066b6106" containerName="main" containerID="cri-o://b0fd8707663361403e095953467ff6b8e9edb6d1ea1f11d9aaa1ca21b4cb4d6a" gracePeriod=45
Sep 18 21:25:27.909789 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:27.909756    2579 generic.go:356] "Generic (PLEG): container finished" podID="0bf98fb9-40a7-4a01-b5c6-99e7066b6106" containerID="b0fd8707663361403e095953467ff6b8e9edb6d1ea1f11d9aaa1ca21b4cb4d6a" exitCode=0
Sep 18 21:25:27.909963 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:27.909829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd" event={"ID":"0bf98fb9-40a7-4a01-b5c6-99e7066b6106","Type":"ContainerDied","Data":"b0fd8707663361403e095953467ff6b8e9edb6d1ea1f11d9aaa1ca21b4cb4d6a"}
Sep 18 21:25:27.977938 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:27.977911    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:25:28.153785 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.153749    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-home\") pod \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") "
Sep 18 21:25:28.153970 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.153835    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hq2g\" (UniqueName: \"kubernetes.io/projected/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-kube-api-access-2hq2g\") pod \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") "
Sep 18 21:25:28.153970 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.153863    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-tls-certs\") pod \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") "
Sep 18 21:25:28.153970 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.153891    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-model-cache\") pod \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") "
Sep 18 21:25:28.153970 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.153944    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-dshm\") pod \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") "
Sep 18 21:25:28.153970 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.153963    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-tmp-dir\") pod \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\" (UID: \"0bf98fb9-40a7-4a01-b5c6-99e7066b6106\") "
Sep 18 21:25:28.154236 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.154075    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-home" (OuterVolumeSpecName: "home") pod "0bf98fb9-40a7-4a01-b5c6-99e7066b6106" (UID: "0bf98fb9-40a7-4a01-b5c6-99e7066b6106"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:25:28.154295 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.154242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bf98fb9-40a7-4a01-b5c6-99e7066b6106" (UID: "0bf98fb9-40a7-4a01-b5c6-99e7066b6106"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:25:28.154362 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.154319    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bf98fb9-40a7-4a01-b5c6-99e7066b6106" (UID: "0bf98fb9-40a7-4a01-b5c6-99e7066b6106"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:25:28.154414 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.154386    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-model-cache\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:28.154414 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.154400    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-home\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:28.156240 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.156214    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bf98fb9-40a7-4a01-b5c6-99e7066b6106" (UID: "0bf98fb9-40a7-4a01-b5c6-99e7066b6106"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:25:28.156350 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.156236    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-dshm" (OuterVolumeSpecName: "dshm") pod "0bf98fb9-40a7-4a01-b5c6-99e7066b6106" (UID: "0bf98fb9-40a7-4a01-b5c6-99e7066b6106"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:25:28.156350 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.156250    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-kube-api-access-2hq2g" (OuterVolumeSpecName: "kube-api-access-2hq2g") pod "0bf98fb9-40a7-4a01-b5c6-99e7066b6106" (UID: "0bf98fb9-40a7-4a01-b5c6-99e7066b6106"). InnerVolumeSpecName "kube-api-access-2hq2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:25:28.255888 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.255855    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hq2g\" (UniqueName: \"kubernetes.io/projected/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-kube-api-access-2hq2g\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:28.255888 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.255883    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-tls-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:28.255888 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.255894    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-dshm\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:28.256094 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.255902    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf98fb9-40a7-4a01-b5c6-99e7066b6106-tmp-dir\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:25:28.899953 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.899913    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:28.916876 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.916850    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"
Sep 18 21:25:28.923538 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.923509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd" event={"ID":"0bf98fb9-40a7-4a01-b5c6-99e7066b6106","Type":"ContainerDied","Data":"e47e566045ddf363ba2a3519537136f8203d6527f1824a023efc2d390e5b66e2"}
Sep 18 21:25:28.923692 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.923549    2579 scope.go:117] "RemoveContainer" containerID="b0fd8707663361403e095953467ff6b8e9edb6d1ea1f11d9aaa1ca21b4cb4d6a"
Sep 18 21:25:28.944729 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.944701    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"]
Sep 18 21:25:28.948857 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:28.948831    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-773e07/explicit-model-773e07-kserve-696544f94c-5xzbd"]
Sep 18 21:25:29.000433 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:29.000401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:29.208117 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:29.208034    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:29.556887 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:29.556859    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:29.663700 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:29.663662    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:29.758584 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:29.758550    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:30.922690 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:30.922628    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bf98fb9-40a7-4a01-b5c6-99e7066b6106" path="/var/lib/kubelet/pods/0bf98fb9-40a7-4a01-b5c6-99e7066b6106/volumes"
Sep 18 21:25:36.271627 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:36.271592    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:36.373436 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:36.373402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:36.512928 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:36.512895    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:36.653940 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:36.653908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:41.786629 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:41.786592    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:43.364365 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:43.364327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:43.479275 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:43.479239    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:44.294693 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:44.294603    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:44.684904 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:44.684817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:44.861779 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:44.861745    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:47.617804 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:47.617766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:48.587883 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:48.587843    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:48.777851 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:48.777815    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:49.644040 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:49.644006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:50.413428 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:50.413395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:50.678125 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:50.678046    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:50.952198 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:50.952110    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:51.531563 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.531528    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"]
Sep 18 21:25:51.532041 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.532024    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="672246a1-47aa-40e0-88a1-123b217d4676" containerName="maas-api"
Sep 18 21:25:51.532087 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.532043    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="672246a1-47aa-40e0-88a1-123b217d4676" containerName="maas-api"
Sep 18 21:25:51.532123 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.532089    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a0c72d-d89f-4feb-a24c-f182fab7ae25" containerName="maas-api"
Sep 18 21:25:51.532123 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.532099    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a0c72d-d89f-4feb-a24c-f182fab7ae25" containerName="maas-api"
Sep 18 21:25:51.532123 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.532109    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf98fb9-40a7-4a01-b5c6-99e7066b6106" containerName="main"
Sep 18 21:25:51.532123 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.532116    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf98fb9-40a7-4a01-b5c6-99e7066b6106" containerName="main"
Sep 18 21:25:51.532250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.532190    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bf98fb9-40a7-4a01-b5c6-99e7066b6106" containerName="main"
Sep 18 21:25:51.532250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.532201    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="672246a1-47aa-40e0-88a1-123b217d4676" containerName="maas-api"
Sep 18 21:25:51.532250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.532208    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="32a0c72d-d89f-4feb-a24c-f182fab7ae25" containerName="maas-api"
Sep 18 21:25:51.535512 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.535493    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:51.539228 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.539202    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-1b65ac-serving-cert\""
Sep 18 21:25:51.567169 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.567140    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"]
Sep 18 21:25:51.593927 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.593895    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:51.688979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.688939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ad9f28-88ec-450e-95fa-0e812c2de534-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:51.688979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.688984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9ad9f28-88ec-450e-95fa-0e812c2de534-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:51.689205 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.689004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbx82\" (UniqueName: \"kubernetes.io/projected/b9ad9f28-88ec-450e-95fa-0e812c2de534-kube-api-access-vbx82\") pod \"maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:51.689205 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.689123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ad9f28-88ec-450e-95fa-0e812c2de534-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:51.749151 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.749110    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:51.790585 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.790550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ad9f28-88ec-450e-95fa-0e812c2de534-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:51.790585 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.790588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9ad9f28-88ec-450e-95fa-0e812c2de534-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:51.790874 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.790608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbx82\" (UniqueName: \"kubernetes.io/projected/b9ad9f28-88ec-450e-95fa-0e812c2de534-kube-api-access-vbx82\") pod \"maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:51.790874 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.790686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ad9f28-88ec-450e-95fa-0e812c2de534-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:51.791306 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.791278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ad9f28-88ec-450e-95fa-0e812c2de534-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:51.793217 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.793192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9ad9f28-88ec-450e-95fa-0e812c2de534-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:51.793331 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.793217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ad9f28-88ec-450e-95fa-0e812c2de534-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:51.808999 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.808971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbx82\" (UniqueName: \"kubernetes.io/projected/b9ad9f28-88ec-450e-95fa-0e812c2de534-kube-api-access-vbx82\") pod \"maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:51.847095 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:51.847071    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:52.125784 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:52.125758    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"]
Sep 18 21:25:52.127677 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:25:52.127627    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ad9f28_88ec_450e_95fa_0e812c2de534.slice/crio-b08a4eb77e608561d4ecb90416e0013092d7099310bc8995ad641b93d38e7e59 WatchSource:0}: Error finding container b08a4eb77e608561d4ecb90416e0013092d7099310bc8995ad641b93d38e7e59: Status 404 returned error can't find the container with id b08a4eb77e608561d4ecb90416e0013092d7099310bc8995ad641b93d38e7e59
Sep 18 21:25:52.291875 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:52.291845    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:52.388682 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:52.388574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:25:53.051829 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:53.051793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh" event={"ID":"b9ad9f28-88ec-450e-95fa-0e812c2de534","Type":"ContainerStarted","Data":"abe8c51ef42f1096eded7dbe412bdc969815547f4f2f2cc97b67631110cef8c4"}
Sep 18 21:25:53.052253 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:53.051850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh" event={"ID":"b9ad9f28-88ec-450e-95fa-0e812c2de534","Type":"ContainerStarted","Data":"b08a4eb77e608561d4ecb90416e0013092d7099310bc8995ad641b93d38e7e59"}
Sep 18 21:25:53.052253 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:53.051899    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:25:53.075656 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:53.075604    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh" podStartSLOduration=1.655024617 podStartE2EDuration="2.07558964s" totalImagesPullingTime="420.565023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:25:53.074205851 +0000 UTC m=+1190.681265411" watchObservedRunningTime="2026-09-18 21:25:53.07558964 +0000 UTC m=+1190.682649198"
Sep 18 21:25:59.062576 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:25:59.062542    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:26:00.423484 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:00.423447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:00.649864 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:00.649827    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"]
Sep 18 21:26:00.650182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:00.650153    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerName="istio-proxy" containerID="cri-o://38b57b49e007b6a83a9e56127b8c1cc7f668cdc7db5c5ed1b7c9980b8e6ae661" gracePeriod=30
Sep 18 21:26:00.687469 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:00.687398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:01.218427 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:01.218391    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:01.582222 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:01.582185    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:01.684450 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:01.684416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:02.743794 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:02.743760    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:02.835758 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:02.835727    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:02.893493 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:02.893464    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:26:02.896796 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:02.896722    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:26:02.897611 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:02.897587    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:26:02.900558 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:02.900542    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:26:02.950758 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:02.950727    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:05.902533 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:05.902507    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:26:06.041480 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.041454    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-envoy\") pod \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") "
Sep 18 21:26:06.041640 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.041517    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7874\" (UniqueName: \"kubernetes.io/projected/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-kube-api-access-h7874\") pod \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") "
Sep 18 21:26:06.041640 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.041570    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-data\") pod \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") "
Sep 18 21:26:06.041640 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.041632    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-credential-socket\") pod \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") "
Sep 18 21:26:06.041814 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.041693    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-workload-socket\") pod \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") "
Sep 18 21:26:06.041814 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.041781    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-podinfo\") pod \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") "
Sep 18 21:26:06.041915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.041816    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-workload-certs\") pod \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") "
Sep 18 21:26:06.041915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.041857    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istiod-ca-cert\") pod \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") "
Sep 18 21:26:06.041915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.041894    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-token\") pod \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\" (UID: \"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac\") "
Sep 18 21:26:06.042065 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.041912    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" (UID: "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:26:06.042065 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.041925    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" (UID: "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:26:06.042065 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.041992    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" (UID: "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:26:06.042204 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.042150    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" (UID: "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:26:06.042331 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.042304    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" (UID: "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:26:06.042416 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.042315    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-credential-socket\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:26:06.042416 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.042359    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-workload-socket\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:26:06.042416 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.042376    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-workload-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:26:06.042416 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.042391    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-data\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:26:06.043806 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.043772    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" (UID: "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:26:06.043806 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.043797    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" (UID: "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 21:26:06.043948 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.043836    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-kube-api-access-h7874" (OuterVolumeSpecName: "kube-api-access-h7874") pod "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" (UID: "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac"). InnerVolumeSpecName "kube-api-access-h7874". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:26:06.044053 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.044039    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" (UID: "a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:26:06.120500 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.120468    2579 generic.go:356] "Generic (PLEG): container finished" podID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerID="38b57b49e007b6a83a9e56127b8c1cc7f668cdc7db5c5ed1b7c9980b8e6ae661" exitCode=0
Sep 18 21:26:06.120682 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.120525    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"
Sep 18 21:26:06.120682 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.120559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" event={"ID":"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac","Type":"ContainerDied","Data":"38b57b49e007b6a83a9e56127b8c1cc7f668cdc7db5c5ed1b7c9980b8e6ae661"}
Sep 18 21:26:06.120682 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.120630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9" event={"ID":"a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac","Type":"ContainerDied","Data":"4748b58eee18a431cb01b54f3a03925d10f471bf98871d939d793d336d1525bf"}
Sep 18 21:26:06.120682 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.120670    2579 scope.go:117] "RemoveContainer" containerID="38b57b49e007b6a83a9e56127b8c1cc7f668cdc7db5c5ed1b7c9980b8e6ae661"
Sep 18 21:26:06.130064 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.130047    2579 scope.go:117] "RemoveContainer" containerID="3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8"
Sep 18 21:26:06.138013 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.137997    2579 scope.go:117] "RemoveContainer" containerID="38b57b49e007b6a83a9e56127b8c1cc7f668cdc7db5c5ed1b7c9980b8e6ae661"
Sep 18 21:26:06.138303 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:26:06.138284    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b57b49e007b6a83a9e56127b8c1cc7f668cdc7db5c5ed1b7c9980b8e6ae661\": container with ID starting with 38b57b49e007b6a83a9e56127b8c1cc7f668cdc7db5c5ed1b7c9980b8e6ae661 not found: ID does not exist" containerID="38b57b49e007b6a83a9e56127b8c1cc7f668cdc7db5c5ed1b7c9980b8e6ae661"
Sep 18 21:26:06.138345 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.138312    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b57b49e007b6a83a9e56127b8c1cc7f668cdc7db5c5ed1b7c9980b8e6ae661"} err="failed to get container status \"38b57b49e007b6a83a9e56127b8c1cc7f668cdc7db5c5ed1b7c9980b8e6ae661\": rpc error: code = NotFound desc = could not find container \"38b57b49e007b6a83a9e56127b8c1cc7f668cdc7db5c5ed1b7c9980b8e6ae661\": container with ID starting with 38b57b49e007b6a83a9e56127b8c1cc7f668cdc7db5c5ed1b7c9980b8e6ae661 not found: ID does not exist"
Sep 18 21:26:06.138345 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.138329    2579 scope.go:117] "RemoveContainer" containerID="3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8"
Sep 18 21:26:06.138589 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:26:06.138571    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8\": container with ID starting with 3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8 not found: ID does not exist" containerID="3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8"
Sep 18 21:26:06.138666 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.138597    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8"} err="failed to get container status \"3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8\": rpc error: code = NotFound desc = could not find container \"3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8\": container with ID starting with 3c1cb85cfe2ee644e9252dbcd0ca597f2352a63e8768ea291068e74aa58299c8 not found: ID does not exist"
Sep 18 21:26:06.143093 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.143073    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-token\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:26:06.143093 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.143092    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-envoy\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:26:06.143216 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.143102    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7874\" (UniqueName: \"kubernetes.io/projected/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-kube-api-access-h7874\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:26:06.143216 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.143110    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istio-podinfo\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:26:06.143216 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.143120    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac-istiod-ca-cert\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:26:06.159063 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.159037    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"]
Sep 18 21:26:06.169513 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.169489    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-773e07-openshift-default-7456b59cd4-q4sm9"]
Sep 18 21:26:06.922566 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:06.922528    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" path="/var/lib/kubelet/pods/a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac/volumes"
Sep 18 21:26:14.865309 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:14.865268    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:15.598779 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:15.598742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:16.541864 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:16.541822    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:17.653131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:17.653093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:18.390308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:18.390274    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:18.488964 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:18.488925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:18.748453 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:18.748363    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:18.992331 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:18.992285    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"]
Sep 18 21:26:18.992825 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:18.992804    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerName="istio-proxy"
Sep 18 21:26:18.992825 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:18.992823    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerName="istio-proxy"
Sep 18 21:26:18.993071 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:18.992842    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerName="istio-proxy"
Sep 18 21:26:18.993071 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:18.992848    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerName="istio-proxy"
Sep 18 21:26:18.993071 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:18.992923    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerName="istio-proxy"
Sep 18 21:26:18.997659 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:18.997616    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.001195 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.001120    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-514a1a-openshift-default-dockercfg-m5mwn\""
Sep 18 21:26:19.001634 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.001615    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 21:26:19.013417 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.013379    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"]
Sep 18 21:26:19.019390 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.019352    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:19.072291 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.072256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-data\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.072487 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.072302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278bea04-a288-4f2b-8686-c444ec8d4b43-istiod-ca-cert\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.072487 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.072366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-workload-certs\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.072487 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.072384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59mqh\" (UniqueName: \"kubernetes.io/projected/278bea04-a288-4f2b-8686-c444ec8d4b43-kube-api-access-59mqh\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.072487 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.072432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-envoy\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.072487 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.072453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-token\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.072487 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.072477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-credential-socket\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.072793 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.072522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-workload-socket\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.072793 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.072542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-podinfo\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.173747 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.173710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-data\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.173747 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.173753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278bea04-a288-4f2b-8686-c444ec8d4b43-istiod-ca-cert\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.174014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.173787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-workload-certs\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.174014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.173899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59mqh\" (UniqueName: \"kubernetes.io/projected/278bea04-a288-4f2b-8686-c444ec8d4b43-kube-api-access-59mqh\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.174014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.173966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-envoy\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.174014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.173999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-token\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.174223 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.174043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-credential-socket\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.174223 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.174126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-workload-socket\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.174223 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.174155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-workload-certs\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.174223 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.174158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-podinfo\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.174495 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.174468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-credential-socket\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.174562 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.174495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-data\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.174562 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.174514    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-workload-socket\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.174562 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.174536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278bea04-a288-4f2b-8686-c444ec8d4b43-istiod-ca-cert\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.176274 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.176257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-envoy\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.176513 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.176491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-podinfo\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.183055 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.183022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-token\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.183172 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.183157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59mqh\" (UniqueName: \"kubernetes.io/projected/278bea04-a288-4f2b-8686-c444ec8d4b43-kube-api-access-59mqh\") pod \"e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") " pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.280717 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.280598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:19.311730 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.311681    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:19.599103 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.599073    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"]
Sep 18 21:26:19.600703 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:26:19.600664    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278bea04_a288_4f2b_8686_c444ec8d4b43.slice/crio-2215ca798a038f5042382b79b5a1036d55bb1cfffdf2e50807601daf99526ed2 WatchSource:0}: Error finding container 2215ca798a038f5042382b79b5a1036d55bb1cfffdf2e50807601daf99526ed2: Status 404 returned error can't find the container with id 2215ca798a038f5042382b79b5a1036d55bb1cfffdf2e50807601daf99526ed2
Sep 18 21:26:19.603206 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.603172    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:26:19.603302 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.603243    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:26:19.603302 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:19.603271    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:26:20.197254 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:20.197213    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" event={"ID":"278bea04-a288-4f2b-8686-c444ec8d4b43","Type":"ContainerStarted","Data":"5d5eb188ecd608a8b31ea7e61a9b56920f867d4862bf2b2167e460b6c79049ea"}
Sep 18 21:26:20.197718 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:20.197269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" event={"ID":"278bea04-a288-4f2b-8686-c444ec8d4b43","Type":"ContainerStarted","Data":"2215ca798a038f5042382b79b5a1036d55bb1cfffdf2e50807601daf99526ed2"}
Sep 18 21:26:20.223302 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:20.223249    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" podStartSLOduration=2.223231825 podStartE2EDuration="2.223231825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:26:20.220813484 +0000 UTC m=+1217.827873053" watchObservedRunningTime="2026-09-18 21:26:20.223231825 +0000 UTC m=+1217.830291444"
Sep 18 21:26:20.312547 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:20.312507    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:20.314272 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:20.314248    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 18 21:26:20.314344 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:20.314314    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 18 21:26:21.317271 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:21.317238    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:21.508695 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:21.508661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:22.194755 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:22.194717    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:22.207231 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:22.207206    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:22.208224 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:22.208202    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:22.313487 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:22.313455    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:22.481199 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:22.481086    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:23.427655 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:23.427620    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:24.397998 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:24.397957    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:25.336579 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:25.336544    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:26.119996 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.119960    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"]
Sep 18 21:26:26.120708 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.120688    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e2a5c8-d9a4-4eb2-a61c-f428eccb56ac" containerName="istio-proxy"
Sep 18 21:26:26.124977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.124953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.130308 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.130284    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-514a1a-serving-cert\""
Sep 18 21:26:26.145582 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.145544    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"]
Sep 18 21:26:26.179246 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.179213    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:26.247250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.247208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frsv9\" (UniqueName: \"kubernetes.io/projected/97187b4f-0659-401e-956a-0da472757be4-kube-api-access-frsv9\") pod \"maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.247438 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.247283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97187b4f-0659-401e-956a-0da472757be4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.247438 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.247314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97187b4f-0659-401e-956a-0da472757be4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.247438 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.247404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97187b4f-0659-401e-956a-0da472757be4-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.274478 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.274434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:26.348049 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.348019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97187b4f-0659-401e-956a-0da472757be4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.348231 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.348059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97187b4f-0659-401e-956a-0da472757be4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.348231 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.348110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97187b4f-0659-401e-956a-0da472757be4-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.348231 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.348161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frsv9\" (UniqueName: \"kubernetes.io/projected/97187b4f-0659-401e-956a-0da472757be4-kube-api-access-frsv9\") pod \"maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.348908 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.348881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97187b4f-0659-401e-956a-0da472757be4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.350508 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.350481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97187b4f-0659-401e-956a-0da472757be4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.350614 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.350569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97187b4f-0659-401e-956a-0da472757be4-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.374280 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.374209    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frsv9\" (UniqueName: \"kubernetes.io/projected/97187b4f-0659-401e-956a-0da472757be4-kube-api-access-frsv9\") pod \"maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.389412 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.389380    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:26.436578 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.436535    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:26.734061 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:26.733974    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"]
Sep 18 21:26:26.738234 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:26:26.738202    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97187b4f_0659_401e_956a_0da472757be4.slice/crio-00fa67b41482702806f8c958f25f5be94e73c6bc3a191433049a64cbb5167759 WatchSource:0}: Error finding container 00fa67b41482702806f8c958f25f5be94e73c6bc3a191433049a64cbb5167759: Status 404 returned error can't find the container with id 00fa67b41482702806f8c958f25f5be94e73c6bc3a191433049a64cbb5167759
Sep 18 21:26:27.122733 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:27.122682    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:27.237953 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:27.237912    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:27.240549 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:27.240517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc" event={"ID":"97187b4f-0659-401e-956a-0da472757be4","Type":"ContainerStarted","Data":"e8cac72c7b437a4f24525f099166c1b146c661885e7806c71fa93b99f92de8c3"}
Sep 18 21:26:27.240721 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:27.240594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc" event={"ID":"97187b4f-0659-401e-956a-0da472757be4","Type":"ContainerStarted","Data":"00fa67b41482702806f8c958f25f5be94e73c6bc3a191433049a64cbb5167759"}
Sep 18 21:26:27.240721 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:27.240668    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:27.280108 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:27.280048    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc" podStartSLOduration=0.93644465 podStartE2EDuration="1.280031733s" totalImagesPullingTime="343.587083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:26:27.278231978 +0000 UTC m=+1224.885291535" watchObservedRunningTime="2026-09-18 21:26:27.280031733 +0000 UTC m=+1224.887091292"
Sep 18 21:26:33.252492 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:33.252462    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:26:38.424568 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:38.424528    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:41.003583 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.003544    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"]
Sep 18 21:26:41.007461 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.007437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.010554 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.010534    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f5a86c\"/\"default-dockercfg-c47rk\""
Sep 18 21:26:41.011566 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.011544    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f5a86c\"/\"noten-model-f5a86c-kserve-self-signed-certs\""
Sep 18 21:26:41.011566 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.011552    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f5a86c\"/\"openshift-service-ca.crt\""
Sep 18 21:26:41.011787 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.011585    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f5a86c\"/\"kube-root-ca.crt\""
Sep 18 21:26:41.018425 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.018400    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"]
Sep 18 21:26:41.039382 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.039349    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:41.102178 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.102144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-dshm\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.102370 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.102183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7gcc\" (UniqueName: \"kubernetes.io/projected/b9615dc5-82ac-4723-8e26-14dae067233b-kube-api-access-k7gcc\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.102370 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.102290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-model-cache\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.102370 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.102349    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9615dc5-82ac-4723-8e26-14dae067233b-tls-certs\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.102543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.102387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-home\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.102592 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.102561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-tmp-dir\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.204225 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.204175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-dshm\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.204405 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.204238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7gcc\" (UniqueName: \"kubernetes.io/projected/b9615dc5-82ac-4723-8e26-14dae067233b-kube-api-access-k7gcc\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.204405 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.204318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-model-cache\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.204405 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.204374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9615dc5-82ac-4723-8e26-14dae067233b-tls-certs\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.204550 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.204413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-home\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.204550 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.204502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-tmp-dir\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.205027 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.204997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-tmp-dir\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.205134 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.205013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-model-cache\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.205134 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.205058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-home\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.206886 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.206864    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-dshm\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.207414 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.207397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9615dc5-82ac-4723-8e26-14dae067233b-tls-certs\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.213969 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.213949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7gcc\" (UniqueName: \"kubernetes.io/projected/b9615dc5-82ac-4723-8e26-14dae067233b-kube-api-access-k7gcc\") pod \"noten-model-f5a86c-kserve-58c6b99b4c-847r9\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") " pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.317869 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.317838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:41.596254 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:41.596226    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"]
Sep 18 21:26:41.597673 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:26:41.597609    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9615dc5_82ac_4723_8e26_14dae067233b.slice/crio-bfcf1fc9e1b7b983eb7d76e0533b72c7196e98c4e6f766eb4c4be86ddbc4beee WatchSource:0}: Error finding container bfcf1fc9e1b7b983eb7d76e0533b72c7196e98c4e6f766eb4c4be86ddbc4beee: Status 404 returned error can't find the container with id bfcf1fc9e1b7b983eb7d76e0533b72c7196e98c4e6f766eb4c4be86ddbc4beee
Sep 18 21:26:42.321940 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:42.321902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9" event={"ID":"b9615dc5-82ac-4723-8e26-14dae067233b","Type":"ContainerStarted","Data":"0025895692b0cd7446aba06d92c4ec6b368340cd6127f0372af6f1ad78ba15e5"}
Sep 18 21:26:42.321940 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:42.321945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9" event={"ID":"b9615dc5-82ac-4723-8e26-14dae067233b","Type":"ContainerStarted","Data":"bfcf1fc9e1b7b983eb7d76e0533b72c7196e98c4e6f766eb4c4be86ddbc4beee"}
Sep 18 21:26:42.362314 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:42.362262    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9" podStartSLOduration=2.362241034 podStartE2EDuration="2.362241034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:26:42.359477081 +0000 UTC m=+1239.966536639" watchObservedRunningTime="2026-09-18 21:26:42.362241034 +0000 UTC m=+1239.969300592"
Sep 18 21:26:44.063336 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:44.063303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:44.564256 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:44.564219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:47.873472 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:47.873438    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"]
Sep 18 21:26:47.894558 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:47.894529    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:48.247697 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:48.247572    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:49.341829 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:49.341726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:50.049856 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:50.049812    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"]
Sep 18 21:26:50.102307 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:50.102259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:50.302860 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:50.302741    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:50.449611 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:50.449575    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:50.547098 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:50.547058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:50.697163 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:50.697068    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:51.219762 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:51.219675    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:51.318775 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:51.318728    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:26:51.573215 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:51.573183    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:52.295026 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:52.294993    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:52.392679 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:52.392630    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:53.166136 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:53.166092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:53.248782 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:53.248739    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 21:26:53.248986 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:53.248842    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 21:26:54.369621 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:54.369590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:54.395950 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:54.395913    2579 generic.go:356] "Generic (PLEG): container finished" podID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerID="5d5eb188ecd608a8b31ea7e61a9b56920f867d4862bf2b2167e460b6c79049ea" exitCode=0
Sep 18 21:26:54.396128 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:54.395981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" event={"ID":"278bea04-a288-4f2b-8686-c444ec8d4b43","Type":"ContainerDied","Data":"5d5eb188ecd608a8b31ea7e61a9b56920f867d4862bf2b2167e460b6c79049ea"}
Sep 18 21:26:54.396494 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:54.396476    2579 scope.go:117] "RemoveContainer" containerID="5d5eb188ecd608a8b31ea7e61a9b56920f867d4862bf2b2167e460b6c79049ea"
Sep 18 21:26:54.397728 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:54.397700    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:26:54.397792 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:54.397765    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:26:54.397829 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:54.397801    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 21:26:54.819682 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:54.819638    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7867-350910"]
Sep 18 21:26:54.822566 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:54.822546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-350910"
Sep 18 21:26:54.826286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:54.826268    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zrg6b\""
Sep 18 21:26:54.833592 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:54.833567    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7867-350910"]
Sep 18 21:26:54.944354 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:54.944317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxskk\" (UniqueName: \"kubernetes.io/projected/52a7a54d-60ac-467b-a4c3-fb6af9de5064-kube-api-access-mxskk\") pod \"test-curl-7867-350910\" (UID: \"52a7a54d-60ac-467b-a4c3-fb6af9de5064\") " pod="opendatahub/test-curl-7867-350910"
Sep 18 21:26:55.045731 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:55.045690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxskk\" (UniqueName: \"kubernetes.io/projected/52a7a54d-60ac-467b-a4c3-fb6af9de5064-kube-api-access-mxskk\") pod \"test-curl-7867-350910\" (UID: \"52a7a54d-60ac-467b-a4c3-fb6af9de5064\") " pod="opendatahub/test-curl-7867-350910"
Sep 18 21:26:55.055075 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:55.055052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxskk\" (UniqueName: \"kubernetes.io/projected/52a7a54d-60ac-467b-a4c3-fb6af9de5064-kube-api-access-mxskk\") pod \"test-curl-7867-350910\" (UID: \"52a7a54d-60ac-467b-a4c3-fb6af9de5064\") " pod="opendatahub/test-curl-7867-350910"
Sep 18 21:26:55.132875 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:55.132789    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-350910"
Sep 18 21:26:55.317574 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:55.317538    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:55.403405 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:55.403315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" event={"ID":"278bea04-a288-4f2b-8686-c444ec8d4b43","Type":"ContainerStarted","Data":"43a35632b87c08bd4a8fc20bbd9b5b9b00fcac848cbb09e92684ac59afdae90a"}
Sep 18 21:26:55.410965 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:55.410941    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7867-350910"]
Sep 18 21:26:55.413021 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:26:55.412990    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52a7a54d_60ac_467b_a4c3_fb6af9de5064.slice/crio-57221ac9b6f32f59243e61b5260f3aaf22ac32e5f6c086cc4a28fc2b2a82861b WatchSource:0}: Error finding container 57221ac9b6f32f59243e61b5260f3aaf22ac32e5f6c086cc4a28fc2b2a82861b: Status 404 returned error can't find the container with id 57221ac9b6f32f59243e61b5260f3aaf22ac32e5f6c086cc4a28fc2b2a82861b
Sep 18 21:26:55.637238 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:55.637209    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:56.274264 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:56.274228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:56.312528 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:56.312481    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:26:56.312841 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:56.312816    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 18 21:26:56.312915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:56.312869    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 18 21:26:56.410989 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:56.410943    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7867-350910" event={"ID":"52a7a54d-60ac-467b-a4c3-fb6af9de5064","Type":"ContainerStarted","Data":"57221ac9b6f32f59243e61b5260f3aaf22ac32e5f6c086cc4a28fc2b2a82861b"}
Sep 18 21:26:57.312462 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:57.312420    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 18 21:26:57.312657 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:57.312485    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 18 21:26:58.230187 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:58.230062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:58.312615 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:58.312576    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 18 21:26:58.312815 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:58.312638    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 18 21:26:58.426482 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:58.426373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7867-350910" event={"ID":"52a7a54d-60ac-467b-a4c3-fb6af9de5064","Type":"ContainerStarted","Data":"49a1b6639558f86be12c7185187e44e68b19afd6f3ef6da596a98ddadf1f3ec4"}
Sep 18 21:26:58.472267 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:58.472210    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7867-350910" podStartSLOduration=1.780078214 podStartE2EDuration="4.472193359s" totalImagesPullingTime="2.692115145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:26:58.46995616 +0000 UTC m=+1256.077015719" watchObservedRunningTime="2026-09-18 21:26:58.472193359 +0000 UTC m=+1256.079253004"
Sep 18 21:26:59.034669 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.034213    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"]
Sep 18 21:26:59.034669 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.034461    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg" podUID="14fc27ce-2b89-43c8-97b7-e2f2807637e2" containerName="maas-api" containerID="cri-o://9a0346f30581cce72a00f9bc13d7c8d1a111187b1f5281fb5fa5860dc84354a3" gracePeriod=30
Sep 18 21:26:59.070056 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.070022    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:59.167265 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.167236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:59.184967 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.184904    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7867-350910"]
Sep 18 21:26:59.197348 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.197305    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7867-350910"]
Sep 18 21:26:59.270050 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.269978    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:59.312079 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.312043    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 18 21:26:59.312250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.312105    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 18 21:26:59.363818 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.363785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:59.376911 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.376887    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:26:59.454091 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.453971    2579 generic.go:356] "Generic (PLEG): container finished" podID="14fc27ce-2b89-43c8-97b7-e2f2807637e2" containerID="9a0346f30581cce72a00f9bc13d7c8d1a111187b1f5281fb5fa5860dc84354a3" exitCode=0
Sep 18 21:26:59.454091 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.454032    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"
Sep 18 21:26:59.454091 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.454067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg" event={"ID":"14fc27ce-2b89-43c8-97b7-e2f2807637e2","Type":"ContainerDied","Data":"9a0346f30581cce72a00f9bc13d7c8d1a111187b1f5281fb5fa5860dc84354a3"}
Sep 18 21:26:59.454339 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.454135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg" event={"ID":"14fc27ce-2b89-43c8-97b7-e2f2807637e2","Type":"ContainerDied","Data":"3ee9eb97be2e5dc0f92b8041a32ef3aacb5a367c3aa9b6806e2fcc51756eb4d7"}
Sep 18 21:26:59.454339 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.454194    2579 scope.go:117] "RemoveContainer" containerID="9a0346f30581cce72a00f9bc13d7c8d1a111187b1f5281fb5fa5860dc84354a3"
Sep 18 21:26:59.454431 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:26:59.454402    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7867-350910" container="test-curl-7867-350910"
Sep 18 21:26:59.464136 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.464066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:59.464548 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.464524    2579 scope.go:117] "RemoveContainer" containerID="9a0346f30581cce72a00f9bc13d7c8d1a111187b1f5281fb5fa5860dc84354a3"
Sep 18 21:26:59.464889 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:26:59.464867    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0346f30581cce72a00f9bc13d7c8d1a111187b1f5281fb5fa5860dc84354a3\": container with ID starting with 9a0346f30581cce72a00f9bc13d7c8d1a111187b1f5281fb5fa5860dc84354a3 not found: ID does not exist" containerID="9a0346f30581cce72a00f9bc13d7c8d1a111187b1f5281fb5fa5860dc84354a3"
Sep 18 21:26:59.464980 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.464900    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0346f30581cce72a00f9bc13d7c8d1a111187b1f5281fb5fa5860dc84354a3"} err="failed to get container status \"9a0346f30581cce72a00f9bc13d7c8d1a111187b1f5281fb5fa5860dc84354a3\": rpc error: code = NotFound desc = could not find container \"9a0346f30581cce72a00f9bc13d7c8d1a111187b1f5281fb5fa5860dc84354a3\": container with ID starting with 9a0346f30581cce72a00f9bc13d7c8d1a111187b1f5281fb5fa5860dc84354a3 not found: ID does not exist"
Sep 18 21:26:59.493921 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.493887    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14fc27ce-2b89-43c8-97b7-e2f2807637e2-maas-api-tls\") pod \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") "
Sep 18 21:26:59.494114 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.494013    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14fc27ce-2b89-43c8-97b7-e2f2807637e2-metrics-certs\") pod \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") "
Sep 18 21:26:59.494114 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.494081    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rjnx\" (UniqueName: \"kubernetes.io/projected/14fc27ce-2b89-43c8-97b7-e2f2807637e2-kube-api-access-9rjnx\") pod \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") "
Sep 18 21:26:59.494242 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.494133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fc27ce-2b89-43c8-97b7-e2f2807637e2-openshift-service-ca\") pod \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\" (UID: \"14fc27ce-2b89-43c8-97b7-e2f2807637e2\") "
Sep 18 21:26:59.494888 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.494856    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14fc27ce-2b89-43c8-97b7-e2f2807637e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "14fc27ce-2b89-43c8-97b7-e2f2807637e2" (UID: "14fc27ce-2b89-43c8-97b7-e2f2807637e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:26:59.497357 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.497268    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14fc27ce-2b89-43c8-97b7-e2f2807637e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "14fc27ce-2b89-43c8-97b7-e2f2807637e2" (UID: "14fc27ce-2b89-43c8-97b7-e2f2807637e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:26:59.497357 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.497310    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14fc27ce-2b89-43c8-97b7-e2f2807637e2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "14fc27ce-2b89-43c8-97b7-e2f2807637e2" (UID: "14fc27ce-2b89-43c8-97b7-e2f2807637e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:26:59.497530 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.497397    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fc27ce-2b89-43c8-97b7-e2f2807637e2-kube-api-access-9rjnx" (OuterVolumeSpecName: "kube-api-access-9rjnx") pod "14fc27ce-2b89-43c8-97b7-e2f2807637e2" (UID: "14fc27ce-2b89-43c8-97b7-e2f2807637e2"). InnerVolumeSpecName "kube-api-access-9rjnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:26:59.595304 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.595262    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14fc27ce-2b89-43c8-97b7-e2f2807637e2-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:26:59.595304 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.595298    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14fc27ce-2b89-43c8-97b7-e2f2807637e2-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:26:59.595304 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.595312    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14fc27ce-2b89-43c8-97b7-e2f2807637e2-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:26:59.595552 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.595324    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rjnx\" (UniqueName: \"kubernetes.io/projected/14fc27ce-2b89-43c8-97b7-e2f2807637e2-kube-api-access-9rjnx\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:26:59.752445 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.752352    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:26:59.778431 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.778392    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"]
Sep 18 21:26:59.780529 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.780504    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-116f55-68d7469686-k4fzg"]
Sep 18 21:26:59.958914 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:26:59.958875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:00.312343 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:00.312300    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 18 21:27:00.312796 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:00.312360    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 18 21:27:00.461291 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:00.461227    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7867-350910" podUID="52a7a54d-60ac-467b-a4c3-fb6af9de5064" containerName="test-curl-7867-350910" containerID="cri-o://49a1b6639558f86be12c7185187e44e68b19afd6f3ef6da596a98ddadf1f3ec4" gracePeriod=2
Sep 18 21:27:00.464053 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:27:00.464022    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7867-350910\" is forbidden: User \"system:node:ip-10-0-130-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-92.ec2.internal' and this object" podUID="52a7a54d-60ac-467b-a4c3-fb6af9de5064" pod="opendatahub/test-curl-7867-350910"
Sep 18 21:27:00.938144 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:00.938101    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14fc27ce-2b89-43c8-97b7-e2f2807637e2" path="/var/lib/kubelet/pods/14fc27ce-2b89-43c8-97b7-e2f2807637e2/volumes"
Sep 18 21:27:01.081370 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:01.081335    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:01.312160 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:01.312120    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 18 21:27:01.312360 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:01.312184    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 18 21:27:02.121147 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:02.121088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:02.316222 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:02.316188    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:27:02.316500 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:02.316472    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:27:02.317072 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:02.317053    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:27:02.649522 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:02.649492    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-350910"
Sep 18 21:27:02.726368 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:02.726278    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxskk\" (UniqueName: \"kubernetes.io/projected/52a7a54d-60ac-467b-a4c3-fb6af9de5064-kube-api-access-mxskk\") pod \"52a7a54d-60ac-467b-a4c3-fb6af9de5064\" (UID: \"52a7a54d-60ac-467b-a4c3-fb6af9de5064\") "
Sep 18 21:27:02.728411 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:02.728385    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a7a54d-60ac-467b-a4c3-fb6af9de5064-kube-api-access-mxskk" (OuterVolumeSpecName: "kube-api-access-mxskk") pod "52a7a54d-60ac-467b-a4c3-fb6af9de5064" (UID: "52a7a54d-60ac-467b-a4c3-fb6af9de5064"). InnerVolumeSpecName "kube-api-access-mxskk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:27:02.821074 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:02.821035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:02.827340 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:02.827315    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxskk\" (UniqueName: \"kubernetes.io/projected/52a7a54d-60ac-467b-a4c3-fb6af9de5064-kube-api-access-mxskk\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:02.893757 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:02.893716    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t" podUID="333efa31-f459-40ce-9a4d-94c2a1fd7ebf" containerName="main" containerID="cri-o://207fb1f8b97ab4b4595cdebcb29a7e1a38186839224224bf1451ae01fc7372c9" gracePeriod=45
Sep 18 21:27:02.923638 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:02.923606    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52a7a54d-60ac-467b-a4c3-fb6af9de5064" path="/var/lib/kubelet/pods/52a7a54d-60ac-467b-a4c3-fb6af9de5064/volumes"
Sep 18 21:27:03.140973 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.140939    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:27:03.230396 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.230311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-tmp-dir\") pod \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") "
Sep 18 21:27:03.230396 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.230374    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-tls-certs\") pod \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") "
Sep 18 21:27:03.230629 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.230402    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-dshm\") pod \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") "
Sep 18 21:27:03.230629 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.230416    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-home\") pod \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") "
Sep 18 21:27:03.230629 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.230443    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-model-cache\") pod \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") "
Sep 18 21:27:03.230629 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.230491    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vjn4\" (UniqueName: \"kubernetes.io/projected/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-kube-api-access-2vjn4\") pod \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\" (UID: \"333efa31-f459-40ce-9a4d-94c2a1fd7ebf\") "
Sep 18 21:27:03.230868 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.230637    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "333efa31-f459-40ce-9a4d-94c2a1fd7ebf" (UID: "333efa31-f459-40ce-9a4d-94c2a1fd7ebf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:27:03.230868 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.230751    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-home" (OuterVolumeSpecName: "home") pod "333efa31-f459-40ce-9a4d-94c2a1fd7ebf" (UID: "333efa31-f459-40ce-9a4d-94c2a1fd7ebf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:27:03.230868 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.230826    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-model-cache" (OuterVolumeSpecName: "model-cache") pod "333efa31-f459-40ce-9a4d-94c2a1fd7ebf" (UID: "333efa31-f459-40ce-9a4d-94c2a1fd7ebf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:27:03.230974 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.230935    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-home\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:03.230974 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.230953    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-model-cache\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:03.230974 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.230968    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-tmp-dir\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:03.232732 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.232709    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-dshm" (OuterVolumeSpecName: "dshm") pod "333efa31-f459-40ce-9a4d-94c2a1fd7ebf" (UID: "333efa31-f459-40ce-9a4d-94c2a1fd7ebf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:27:03.232849 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.232832    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "333efa31-f459-40ce-9a4d-94c2a1fd7ebf" (UID: "333efa31-f459-40ce-9a4d-94c2a1fd7ebf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:27:03.232901 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.232855    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-kube-api-access-2vjn4" (OuterVolumeSpecName: "kube-api-access-2vjn4") pod "333efa31-f459-40ce-9a4d-94c2a1fd7ebf" (UID: "333efa31-f459-40ce-9a4d-94c2a1fd7ebf"). InnerVolumeSpecName "kube-api-access-2vjn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:27:03.331699 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.331624    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vjn4\" (UniqueName: \"kubernetes.io/projected/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-kube-api-access-2vjn4\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:03.331699 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.331693    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-tls-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:03.331699 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.331705    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/333efa31-f459-40ce-9a4d-94c2a1fd7ebf-dshm\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:03.480344 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.480312    2579 generic.go:356] "Generic (PLEG): container finished" podID="333efa31-f459-40ce-9a4d-94c2a1fd7ebf" containerID="207fb1f8b97ab4b4595cdebcb29a7e1a38186839224224bf1451ae01fc7372c9" exitCode=0
Sep 18 21:27:03.480536 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.480433    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"
Sep 18 21:27:03.480536 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.480457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t" event={"ID":"333efa31-f459-40ce-9a4d-94c2a1fd7ebf","Type":"ContainerDied","Data":"207fb1f8b97ab4b4595cdebcb29a7e1a38186839224224bf1451ae01fc7372c9"}
Sep 18 21:27:03.480536 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.480517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t" event={"ID":"333efa31-f459-40ce-9a4d-94c2a1fd7ebf","Type":"ContainerDied","Data":"271eb8850eed110ae8ff00a5296a20d5bacd0803a0ccd3db52c39a841959851a"}
Sep 18 21:27:03.480724 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.480543    2579 scope.go:117] "RemoveContainer" containerID="207fb1f8b97ab4b4595cdebcb29a7e1a38186839224224bf1451ae01fc7372c9"
Sep 18 21:27:03.482509 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.482172    2579 generic.go:356] "Generic (PLEG): container finished" podID="52a7a54d-60ac-467b-a4c3-fb6af9de5064" containerID="49a1b6639558f86be12c7185187e44e68b19afd6f3ef6da596a98ddadf1f3ec4" exitCode=137
Sep 18 21:27:03.482509 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.482259    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7867-350910"
Sep 18 21:27:03.491155 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.491139    2579 scope.go:117] "RemoveContainer" containerID="207fb1f8b97ab4b4595cdebcb29a7e1a38186839224224bf1451ae01fc7372c9"
Sep 18 21:27:03.491448 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:27:03.491429    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"207fb1f8b97ab4b4595cdebcb29a7e1a38186839224224bf1451ae01fc7372c9\": container with ID starting with 207fb1f8b97ab4b4595cdebcb29a7e1a38186839224224bf1451ae01fc7372c9 not found: ID does not exist" containerID="207fb1f8b97ab4b4595cdebcb29a7e1a38186839224224bf1451ae01fc7372c9"
Sep 18 21:27:03.491497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.491457    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"207fb1f8b97ab4b4595cdebcb29a7e1a38186839224224bf1451ae01fc7372c9"} err="failed to get container status \"207fb1f8b97ab4b4595cdebcb29a7e1a38186839224224bf1451ae01fc7372c9\": rpc error: code = NotFound desc = could not find container \"207fb1f8b97ab4b4595cdebcb29a7e1a38186839224224bf1451ae01fc7372c9\": container with ID starting with 207fb1f8b97ab4b4595cdebcb29a7e1a38186839224224bf1451ae01fc7372c9 not found: ID does not exist"
Sep 18 21:27:03.491497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.491477    2579 scope.go:117] "RemoveContainer" containerID="49a1b6639558f86be12c7185187e44e68b19afd6f3ef6da596a98ddadf1f3ec4"
Sep 18 21:27:03.500219 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.500202    2579 scope.go:117] "RemoveContainer" containerID="49a1b6639558f86be12c7185187e44e68b19afd6f3ef6da596a98ddadf1f3ec4"
Sep 18 21:27:03.500497 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:27:03.500478    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a1b6639558f86be12c7185187e44e68b19afd6f3ef6da596a98ddadf1f3ec4\": container with ID starting with 49a1b6639558f86be12c7185187e44e68b19afd6f3ef6da596a98ddadf1f3ec4 not found: ID does not exist" containerID="49a1b6639558f86be12c7185187e44e68b19afd6f3ef6da596a98ddadf1f3ec4"
Sep 18 21:27:03.500557 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.500506    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a1b6639558f86be12c7185187e44e68b19afd6f3ef6da596a98ddadf1f3ec4"} err="failed to get container status \"49a1b6639558f86be12c7185187e44e68b19afd6f3ef6da596a98ddadf1f3ec4\": rpc error: code = NotFound desc = could not find container \"49a1b6639558f86be12c7185187e44e68b19afd6f3ef6da596a98ddadf1f3ec4\": container with ID starting with 49a1b6639558f86be12c7185187e44e68b19afd6f3ef6da596a98ddadf1f3ec4 not found: ID does not exist"
Sep 18 21:27:03.507842 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.507819    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"]
Sep 18 21:27:03.515052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:03.515030    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1f86a8/test-model-1f86a8-kserve-77d7d47dfc-nkn8t"]
Sep 18 21:27:04.032547 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.032498    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:04.865105 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.865057    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"]
Sep 18 21:27:04.865531 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.865518    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="333efa31-f459-40ce-9a4d-94c2a1fd7ebf" containerName="main"
Sep 18 21:27:04.865531 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.865532    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="333efa31-f459-40ce-9a4d-94c2a1fd7ebf" containerName="main"
Sep 18 21:27:04.865605 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.865551    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a7a54d-60ac-467b-a4c3-fb6af9de5064" containerName="test-curl-7867-350910"
Sep 18 21:27:04.865605 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.865557    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a7a54d-60ac-467b-a4c3-fb6af9de5064" containerName="test-curl-7867-350910"
Sep 18 21:27:04.865605 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.865569    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14fc27ce-2b89-43c8-97b7-e2f2807637e2" containerName="maas-api"
Sep 18 21:27:04.865605 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.865575    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="14fc27ce-2b89-43c8-97b7-e2f2807637e2" containerName="maas-api"
Sep 18 21:27:04.865759 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.865633    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="333efa31-f459-40ce-9a4d-94c2a1fd7ebf" containerName="main"
Sep 18 21:27:04.865759 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.865654    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="14fc27ce-2b89-43c8-97b7-e2f2807637e2" containerName="maas-api"
Sep 18 21:27:04.865759 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.865661    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="52a7a54d-60ac-467b-a4c3-fb6af9de5064" containerName="test-curl-7867-350910"
Sep 18 21:27:04.873043 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.873021    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:04.885360 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.885337    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6ad49b\"/\"default-dockercfg-97mfm\""
Sep 18 21:27:04.885515 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.885416    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6ad49b\"/\"distinct-w4-6ad49b-kserve-self-signed-certs\""
Sep 18 21:27:04.885515 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.885444    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6ad49b\"/\"kube-root-ca.crt\""
Sep 18 21:27:04.885515 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.885444    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6ad49b\"/\"openshift-service-ca.crt\""
Sep 18 21:27:04.900907 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.900881    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"]
Sep 18 21:27:04.922660 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.922617    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="333efa31-f459-40ce-9a4d-94c2a1fd7ebf" path="/var/lib/kubelet/pods/333efa31-f459-40ce-9a4d-94c2a1fd7ebf/volumes"
Sep 18 21:27:04.948619 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.948589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-model-cache\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:04.948846 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.948633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-tls-certs\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:04.948846 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.948778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-home\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:04.948986 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.948852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-dshm\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:04.948986 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.948915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-tmp-dir\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:04.949069 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.949002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zx9c\" (UniqueName: \"kubernetes.io/projected/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-kube-api-access-7zx9c\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:04.960974 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:04.960935    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:05.050211 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.050179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-model-cache\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:05.050211 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.050216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-tls-certs\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:05.050470 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.050258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-home\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:05.050470 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.050294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-dshm\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:05.050470 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.050340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-tmp-dir\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:05.050470 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.050366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zx9c\" (UniqueName: \"kubernetes.io/projected/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-kube-api-access-7zx9c\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:05.050774 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.050720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-model-cache\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:05.050774 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.050748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-home\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:05.050887 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.050801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-tmp-dir\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:05.052621 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.052601    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-dshm\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:05.052814 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.052795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-tls-certs\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:05.065471 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.065447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zx9c\" (UniqueName: \"kubernetes.io/projected/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-kube-api-access-7zx9c\") pod \"distinct-w4-6ad49b-kserve-5f6976648f-mfvhz\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") " pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:05.069713 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.069658    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9" podUID="b9615dc5-82ac-4723-8e26-14dae067233b" containerName="main" containerID="cri-o://0025895692b0cd7446aba06d92c4ec6b368340cd6127f0372af6f1ad78ba15e5" gracePeriod=45
Sep 18 21:27:05.181984 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.181909    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:05.339260 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.339230    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:27:05.454194 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.454110    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7gcc\" (UniqueName: \"kubernetes.io/projected/b9615dc5-82ac-4723-8e26-14dae067233b-kube-api-access-k7gcc\") pod \"b9615dc5-82ac-4723-8e26-14dae067233b\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") "
Sep 18 21:27:05.454194 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.454149    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-model-cache\") pod \"b9615dc5-82ac-4723-8e26-14dae067233b\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") "
Sep 18 21:27:05.454194 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.454178    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-home\") pod \"b9615dc5-82ac-4723-8e26-14dae067233b\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") "
Sep 18 21:27:05.454465 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.454224    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-tmp-dir\") pod \"b9615dc5-82ac-4723-8e26-14dae067233b\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") "
Sep 18 21:27:05.454465 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.454251    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9615dc5-82ac-4723-8e26-14dae067233b-tls-certs\") pod \"b9615dc5-82ac-4723-8e26-14dae067233b\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") "
Sep 18 21:27:05.454465 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.454291    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-dshm\") pod \"b9615dc5-82ac-4723-8e26-14dae067233b\" (UID: \"b9615dc5-82ac-4723-8e26-14dae067233b\") "
Sep 18 21:27:05.454465 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.454415    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9615dc5-82ac-4723-8e26-14dae067233b" (UID: "b9615dc5-82ac-4723-8e26-14dae067233b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:27:05.454635 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.454499    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9615dc5-82ac-4723-8e26-14dae067233b" (UID: "b9615dc5-82ac-4723-8e26-14dae067233b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:27:05.454635 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.454511    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-home" (OuterVolumeSpecName: "home") pod "b9615dc5-82ac-4723-8e26-14dae067233b" (UID: "b9615dc5-82ac-4723-8e26-14dae067233b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:27:05.454743 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.454685    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-tmp-dir\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:05.454743 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.454700    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-model-cache\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:05.454743 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.454710    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-home\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:05.457019 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.456963    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9615dc5-82ac-4723-8e26-14dae067233b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9615dc5-82ac-4723-8e26-14dae067233b" (UID: "b9615dc5-82ac-4723-8e26-14dae067233b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:27:05.457019 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.456976    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9615dc5-82ac-4723-8e26-14dae067233b-kube-api-access-k7gcc" (OuterVolumeSpecName: "kube-api-access-k7gcc") pod "b9615dc5-82ac-4723-8e26-14dae067233b" (UID: "b9615dc5-82ac-4723-8e26-14dae067233b"). InnerVolumeSpecName "kube-api-access-k7gcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:27:05.457280 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.457256    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-dshm" (OuterVolumeSpecName: "dshm") pod "b9615dc5-82ac-4723-8e26-14dae067233b" (UID: "b9615dc5-82ac-4723-8e26-14dae067233b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:27:05.459459 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.459439    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"]
Sep 18 21:27:05.461564 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:27:05.461543    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ce86f35_5681_4b5a_a9b5_fb706f8acb9c.slice/crio-bb35c514c7333fa997f0a49310cd09222531673e055ec725fd91b65e58706964 WatchSource:0}: Error finding container bb35c514c7333fa997f0a49310cd09222531673e055ec725fd91b65e58706964: Status 404 returned error can't find the container with id bb35c514c7333fa997f0a49310cd09222531673e055ec725fd91b65e58706964
Sep 18 21:27:05.497859 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.497823    2579 generic.go:356] "Generic (PLEG): container finished" podID="b9615dc5-82ac-4723-8e26-14dae067233b" containerID="0025895692b0cd7446aba06d92c4ec6b368340cd6127f0372af6f1ad78ba15e5" exitCode=0
Sep 18 21:27:05.498046 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.497878    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"
Sep 18 21:27:05.498046 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.497916    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9" event={"ID":"b9615dc5-82ac-4723-8e26-14dae067233b","Type":"ContainerDied","Data":"0025895692b0cd7446aba06d92c4ec6b368340cd6127f0372af6f1ad78ba15e5"}
Sep 18 21:27:05.498046 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.497956    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9" event={"ID":"b9615dc5-82ac-4723-8e26-14dae067233b","Type":"ContainerDied","Data":"bfcf1fc9e1b7b983eb7d76e0533b72c7196e98c4e6f766eb4c4be86ddbc4beee"}
Sep 18 21:27:05.498046 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.497981    2579 scope.go:117] "RemoveContainer" containerID="0025895692b0cd7446aba06d92c4ec6b368340cd6127f0372af6f1ad78ba15e5"
Sep 18 21:27:05.499815 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.499788    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz" event={"ID":"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c","Type":"ContainerStarted","Data":"bb35c514c7333fa997f0a49310cd09222531673e055ec725fd91b65e58706964"}
Sep 18 21:27:05.507185 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.507167    2579 scope.go:117] "RemoveContainer" containerID="0025895692b0cd7446aba06d92c4ec6b368340cd6127f0372af6f1ad78ba15e5"
Sep 18 21:27:05.510414 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:27:05.510383    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0025895692b0cd7446aba06d92c4ec6b368340cd6127f0372af6f1ad78ba15e5\": container with ID starting with 0025895692b0cd7446aba06d92c4ec6b368340cd6127f0372af6f1ad78ba15e5 not found: ID does not exist" containerID="0025895692b0cd7446aba06d92c4ec6b368340cd6127f0372af6f1ad78ba15e5"
Sep 18 21:27:05.510525 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.510422    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0025895692b0cd7446aba06d92c4ec6b368340cd6127f0372af6f1ad78ba15e5"} err="failed to get container status \"0025895692b0cd7446aba06d92c4ec6b368340cd6127f0372af6f1ad78ba15e5\": rpc error: code = NotFound desc = could not find container \"0025895692b0cd7446aba06d92c4ec6b368340cd6127f0372af6f1ad78ba15e5\": container with ID starting with 0025895692b0cd7446aba06d92c4ec6b368340cd6127f0372af6f1ad78ba15e5 not found: ID does not exist"
Sep 18 21:27:05.534672 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.534632    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"]
Sep 18 21:27:05.541443 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.541420    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-f5a86c/noten-model-f5a86c-kserve-58c6b99b4c-847r9"]
Sep 18 21:27:05.555530 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.555506    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9615dc5-82ac-4723-8e26-14dae067233b-tls-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:05.555530 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.555530    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9615dc5-82ac-4723-8e26-14dae067233b-dshm\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:05.555693 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:05.555539    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7gcc\" (UniqueName: \"kubernetes.io/projected/b9615dc5-82ac-4723-8e26-14dae067233b-kube-api-access-k7gcc\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:06.359534 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:06.359447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:06.506599 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:06.506560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz" event={"ID":"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c","Type":"ContainerStarted","Data":"66dfc4f2ccad12e63d977a3072d2a8d94e6f83cbfca6e550a3d8108a5f0bff2f"}
Sep 18 21:27:06.549300 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:06.549240    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz" podStartSLOduration=2.549222937 podStartE2EDuration="2.549222937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:27:06.546187213 +0000 UTC m=+1264.153246765" watchObservedRunningTime="2026-09-18 21:27:06.549222937 +0000 UTC m=+1264.156282494"
Sep 18 21:27:06.810923 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:06.810892    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:06.922659 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:06.922617    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9615dc5-82ac-4723-8e26-14dae067233b" path="/var/lib/kubelet/pods/b9615dc5-82ac-4723-8e26-14dae067233b/volumes"
Sep 18 21:27:07.158316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:07.158231    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:07.382329 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:07.382293    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:07.490484 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:07.490384    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:07.663962 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:07.663933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:08.302102 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:08.302070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:09.488339 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:09.488302    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:10.022618 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:10.022590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:10.121662 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:10.121602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:10.632836 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:10.632797    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:11.643887 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:11.643853    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:11.738993 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:11.738955    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:12.624171 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:12.624136    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:13.401639 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.401605    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"]
Sep 18 21:27:13.402393 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.402368    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9615dc5-82ac-4723-8e26-14dae067233b" containerName="main"
Sep 18 21:27:13.402497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.402395    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9615dc5-82ac-4723-8e26-14dae067233b" containerName="main"
Sep 18 21:27:13.402553 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.402510    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9615dc5-82ac-4723-8e26-14dae067233b" containerName="main"
Sep 18 21:27:13.408657 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.408618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:13.411358 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.411332    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-67ccba3b-serving-cert\""
Sep 18 21:27:13.425091 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.425064    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"]
Sep 18 21:27:13.453135 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.453102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:13.536561 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.536530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbddfa6-bb28-42e9-b377-a81fb1ced130-metrics-certs\") pod \"maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:13.536750 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.536583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrps\" (UniqueName: \"kubernetes.io/projected/abbddfa6-bb28-42e9-b377-a81fb1ced130-kube-api-access-fwrps\") pod \"maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:13.536750 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.536738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbddfa6-bb28-42e9-b377-a81fb1ced130-maas-api-tls\") pod \"maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:13.536865 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.536780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbddfa6-bb28-42e9-b377-a81fb1ced130-openshift-service-ca\") pod \"maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:13.557467 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.557432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:13.637728 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.637694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbddfa6-bb28-42e9-b377-a81fb1ced130-metrics-certs\") pod \"maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:13.637939 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.637781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrps\" (UniqueName: \"kubernetes.io/projected/abbddfa6-bb28-42e9-b377-a81fb1ced130-kube-api-access-fwrps\") pod \"maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:13.637939 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.637877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbddfa6-bb28-42e9-b377-a81fb1ced130-maas-api-tls\") pod \"maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:13.637939 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.637921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbddfa6-bb28-42e9-b377-a81fb1ced130-openshift-service-ca\") pod \"maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:13.638578 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.638553    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbddfa6-bb28-42e9-b377-a81fb1ced130-openshift-service-ca\") pod \"maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:13.640251 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.640232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbddfa6-bb28-42e9-b377-a81fb1ced130-metrics-certs\") pod \"maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:13.640347 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.640280    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbddfa6-bb28-42e9-b377-a81fb1ced130-maas-api-tls\") pod \"maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:13.646426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.646402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrps\" (UniqueName: \"kubernetes.io/projected/abbddfa6-bb28-42e9-b377-a81fb1ced130-kube-api-access-fwrps\") pod \"maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:13.703128 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.703021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:13.719094 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:13.719069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:14.016206 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:14.016179    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"]
Sep 18 21:27:14.019023 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:27:14.018996    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbddfa6_bb28_42e9_b377_a81fb1ced130.slice/crio-5f83f7ff5d1e387e7e09bf9c76a3657c85c260d5759886d81436d72d55f67649 WatchSource:0}: Error finding container 5f83f7ff5d1e387e7e09bf9c76a3657c85c260d5759886d81436d72d55f67649: Status 404 returned error can't find the container with id 5f83f7ff5d1e387e7e09bf9c76a3657c85c260d5759886d81436d72d55f67649
Sep 18 21:27:14.248217 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:14.248175    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:14.360196 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:14.360162    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:14.555026 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:14.554989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5" event={"ID":"abbddfa6-bb28-42e9-b377-a81fb1ced130","Type":"ContainerStarted","Data":"943c32d646b16285abcb722c1c435869d14f5c335d89a2b9ac759783a673c8e0"}
Sep 18 21:27:14.555422 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:14.555079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5" event={"ID":"abbddfa6-bb28-42e9-b377-a81fb1ced130","Type":"ContainerStarted","Data":"5f83f7ff5d1e387e7e09bf9c76a3657c85c260d5759886d81436d72d55f67649"}
Sep 18 21:27:14.555422 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:14.555104    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:14.632216 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:14.632103    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5" podStartSLOduration=1.212276476 podStartE2EDuration="1.632086699s" totalImagesPullingTime="419.810223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:27:14.63116581 +0000 UTC m=+1272.238225369" watchObservedRunningTime="2026-09-18 21:27:14.632086699 +0000 UTC m=+1272.239146257"
Sep 18 21:27:15.166356 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:15.166321    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:15.182846 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:15.182819    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:15.182942 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:15.182862    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:15.195454 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:15.195428    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:15.573414 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:15.573382    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:27:16.308042 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:16.308002    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:17.494274 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:17.494234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:18.354804 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:18.354773    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:19.103984 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.103949    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"]
Sep 18 21:27:19.104432 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.104174    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9" podUID="44824cba-e6f7-401d-a981-23c66c97d895" containerName="maas-api" containerID="cri-o://00ae26f9f9f21e7af25e2cfde362c247c12e47f08e32c313e7c0ed861e8f3864" gracePeriod=30
Sep 18 21:27:19.128109 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.128056    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:19.221039 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.221003    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:19.299490 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.299449    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:19.358458 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.358380    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:27:19.437069 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.437023    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:19.500271 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.500241    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x966t\" (UniqueName: \"kubernetes.io/projected/44824cba-e6f7-401d-a981-23c66c97d895-kube-api-access-x966t\") pod \"44824cba-e6f7-401d-a981-23c66c97d895\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") "
Sep 18 21:27:19.500439 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.500294    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-maas-api-tls\") pod \"44824cba-e6f7-401d-a981-23c66c97d895\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") "
Sep 18 21:27:19.500439 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.500315    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-metrics-certs\") pod \"44824cba-e6f7-401d-a981-23c66c97d895\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") "
Sep 18 21:27:19.500577 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.500441    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44824cba-e6f7-401d-a981-23c66c97d895-openshift-service-ca\") pod \"44824cba-e6f7-401d-a981-23c66c97d895\" (UID: \"44824cba-e6f7-401d-a981-23c66c97d895\") "
Sep 18 21:27:19.500866 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.500840    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44824cba-e6f7-401d-a981-23c66c97d895-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "44824cba-e6f7-401d-a981-23c66c97d895" (UID: "44824cba-e6f7-401d-a981-23c66c97d895"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:27:19.502536 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.502510    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "44824cba-e6f7-401d-a981-23c66c97d895" (UID: "44824cba-e6f7-401d-a981-23c66c97d895"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:27:19.502536 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.502528    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44824cba-e6f7-401d-a981-23c66c97d895-kube-api-access-x966t" (OuterVolumeSpecName: "kube-api-access-x966t") pod "44824cba-e6f7-401d-a981-23c66c97d895" (UID: "44824cba-e6f7-401d-a981-23c66c97d895"). InnerVolumeSpecName "kube-api-access-x966t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:27:19.502732 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.502536    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "44824cba-e6f7-401d-a981-23c66c97d895" (UID: "44824cba-e6f7-401d-a981-23c66c97d895"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:27:19.544061 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.544014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:19.585236 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.585169    2579 generic.go:356] "Generic (PLEG): container finished" podID="44824cba-e6f7-401d-a981-23c66c97d895" containerID="00ae26f9f9f21e7af25e2cfde362c247c12e47f08e32c313e7c0ed861e8f3864" exitCode=0
Sep 18 21:27:19.585407 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.585239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9" event={"ID":"44824cba-e6f7-401d-a981-23c66c97d895","Type":"ContainerDied","Data":"00ae26f9f9f21e7af25e2cfde362c247c12e47f08e32c313e7c0ed861e8f3864"}
Sep 18 21:27:19.585407 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.585221    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"
Sep 18 21:27:19.585407 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.585295    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9" event={"ID":"44824cba-e6f7-401d-a981-23c66c97d895","Type":"ContainerDied","Data":"d9ccd6feb9e0202c8a8a41dfb79f18988c03728d7a7522441a7ca01433f8e0f4"}
Sep 18 21:27:19.585407 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.585319    2579 scope.go:117] "RemoveContainer" containerID="00ae26f9f9f21e7af25e2cfde362c247c12e47f08e32c313e7c0ed861e8f3864"
Sep 18 21:27:19.595533 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.595515    2579 scope.go:117] "RemoveContainer" containerID="00ae26f9f9f21e7af25e2cfde362c247c12e47f08e32c313e7c0ed861e8f3864"
Sep 18 21:27:19.595845 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:27:19.595825    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ae26f9f9f21e7af25e2cfde362c247c12e47f08e32c313e7c0ed861e8f3864\": container with ID starting with 00ae26f9f9f21e7af25e2cfde362c247c12e47f08e32c313e7c0ed861e8f3864 not found: ID does not exist" containerID="00ae26f9f9f21e7af25e2cfde362c247c12e47f08e32c313e7c0ed861e8f3864"
Sep 18 21:27:19.595886 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.595854    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ae26f9f9f21e7af25e2cfde362c247c12e47f08e32c313e7c0ed861e8f3864"} err="failed to get container status \"00ae26f9f9f21e7af25e2cfde362c247c12e47f08e32c313e7c0ed861e8f3864\": rpc error: code = NotFound desc = could not find container \"00ae26f9f9f21e7af25e2cfde362c247c12e47f08e32c313e7c0ed861e8f3864\": container with ID starting with 00ae26f9f9f21e7af25e2cfde362c247c12e47f08e32c313e7c0ed861e8f3864 not found: ID does not exist"
Sep 18 21:27:19.602228 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.602205    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:19.602228 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.602227    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44824cba-e6f7-401d-a981-23c66c97d895-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:19.602385 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.602237    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44824cba-e6f7-401d-a981-23c66c97d895-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:19.602385 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.602246    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x966t\" (UniqueName: \"kubernetes.io/projected/44824cba-e6f7-401d-a981-23c66c97d895-kube-api-access-x966t\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:27:19.609935 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.609875    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"]
Sep 18 21:27:19.611770 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.611749    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f86a8-8c6dfd799-gnvj9"]
Sep 18 21:27:19.811690 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:19.811625    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:20.135967 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:20.135928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:20.566160 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:20.566123    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:27:20.922237 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:20.922141    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44824cba-e6f7-401d-a981-23c66c97d895" path="/var/lib/kubelet/pods/44824cba-e6f7-401d-a981-23c66c97d895/volumes"
Sep 18 21:27:21.035179 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:21.035144    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:21.907263 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:21.907226    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:22.661306 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:22.661257    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:23.844207 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:23.844170    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:26.139671 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:26.139615    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:27.420483 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:27.420443    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:27.994966 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:27.994923    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:28.100284 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:28.100244    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:28.308162 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:28.308130    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:28.412399 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:28.412365    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:28.888193 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:28.888159    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:29.149727 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:29.149622    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:29.307500 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:29.307464    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:29.553618 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:29.553584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:29.771374 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:29.771338    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:30.152836 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:30.152799    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:31.498738 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:31.498704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:32.063777 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:32.063742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:32.168598 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:32.168565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:32.605273 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:32.605225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:33.637920 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:33.637875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:34.538682 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:34.538615    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:34.753461 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:34.753392    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:35.730373 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:35.730331    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:36.826545 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.826512    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"]
Sep 18 21:27:36.827142 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.827122    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44824cba-e6f7-401d-a981-23c66c97d895" containerName="maas-api"
Sep 18 21:27:36.827228 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.827147    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="44824cba-e6f7-401d-a981-23c66c97d895" containerName="maas-api"
Sep 18 21:27:36.827310 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.827297    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="44824cba-e6f7-401d-a981-23c66c97d895" containerName="maas-api"
Sep 18 21:27:36.830824 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.830801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:36.839334 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.839312    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-cad5cf-serving-cert\""
Sep 18 21:27:36.859811 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.859779    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"]
Sep 18 21:27:36.878008 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.877959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:36.878189 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.878108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7635a1a0-9bd8-483d-a59e-c89459fbf72a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:36.878189 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.878154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:36.878189 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.878177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st2dk\" (UniqueName: \"kubernetes.io/projected/7635a1a0-9bd8-483d-a59e-c89459fbf72a-kube-api-access-st2dk\") pod \"maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:36.979228 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.979185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st2dk\" (UniqueName: \"kubernetes.io/projected/7635a1a0-9bd8-483d-a59e-c89459fbf72a-kube-api-access-st2dk\") pod \"maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:36.979434 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.979292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:36.979434 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.979379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7635a1a0-9bd8-483d-a59e-c89459fbf72a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:36.979434 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.979426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:36.979614 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:27:36.979442    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-serving-cert: secret "maas-api-e2e-shared-a-w3-cad5cf-serving-cert" not found
Sep 18 21:27:36.979614 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:27:36.979512    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-maas-api-tls podName:7635a1a0-9bd8-483d-a59e-c89459fbf72a nodeName:}" failed. No retries permitted until 2026-09-18 21:27:37.479494492 +0000 UTC m=+1295.086554028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-maas-api-tls") pod "maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl" (UID: "7635a1a0-9bd8-483d-a59e-c89459fbf72a") : secret "maas-api-e2e-shared-a-w3-cad5cf-serving-cert" not found
Sep 18 21:27:36.980039 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.980019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7635a1a0-9bd8-483d-a59e-c89459fbf72a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:36.981954 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:36.981933    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:37.002269 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:37.002246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st2dk\" (UniqueName: \"kubernetes.io/projected/7635a1a0-9bd8-483d-a59e-c89459fbf72a-kube-api-access-st2dk\") pod \"maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:37.014244 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:37.014219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:37.152515 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:37.152434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:37.486250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:37.486141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:37.488603 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:37.488577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:37.617950 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:37.617915    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:37.716310 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:37.716276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:37.740945 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:37.740869    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:38.022849 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:38.022815    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"]
Sep 18 21:27:38.029330 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:27:38.029299    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7635a1a0_9bd8_483d_a59e_c89459fbf72a.slice/crio-b45166a922fccba71f49245ff951d3f59eb6a318c032e9bbf1010f0636124db8 WatchSource:0}: Error finding container b45166a922fccba71f49245ff951d3f59eb6a318c032e9bbf1010f0636124db8: Status 404 returned error can't find the container with id b45166a922fccba71f49245ff951d3f59eb6a318c032e9bbf1010f0636124db8
Sep 18 21:27:38.030867 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:38.030847    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:27:38.691493 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:38.691396    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl" event={"ID":"7635a1a0-9bd8-483d-a59e-c89459fbf72a","Type":"ContainerStarted","Data":"3ed87ecfc5984c3de0c65163d3ab918537694f1898c8e32a5d1d4e528ba6f9ec"}
Sep 18 21:27:38.691493 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:38.691454    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl" event={"ID":"7635a1a0-9bd8-483d-a59e-c89459fbf72a","Type":"ContainerStarted","Data":"b45166a922fccba71f49245ff951d3f59eb6a318c032e9bbf1010f0636124db8"}
Sep 18 21:27:38.691751 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:38.691527    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:38.771296 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:38.771231    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl" podStartSLOduration=2.475713869 podStartE2EDuration="2.771208254s" totalImagesPullingTime="295.494385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:27:38.768174 +0000 UTC m=+1296.375233572" watchObservedRunningTime="2026-09-18 21:27:38.771208254 +0000 UTC m=+1296.378267813"
Sep 18 21:27:39.464356 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:39.464321    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:40.331713 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:40.331678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:43.214495 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:43.214463    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:27:44.702809 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:44.702774    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:27:55.747554 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:27:55.747521    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:02.266849 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:02.266788    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:02.611218 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:02.611183    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:02.796655 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:02.796605    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:02.938531 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:02.938434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:03.142570 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:03.142535    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:03.522444 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:03.522409    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:04.353875 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:04.353836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:05.283201 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:05.283158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:06.314823 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:06.314785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:06.547725 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:06.547693    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:06.789980 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:06.789946    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:07.746014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:07.745976    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:08.820777 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:08.820745    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"]
Sep 18 21:28:08.825713 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:08.825694    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:08.829239 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:08.829216    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-fd993e-serving-cert\""
Sep 18 21:28:08.855930 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:08.855898    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"]
Sep 18 21:28:08.913374 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:08.913340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a60318f-3709-4820-890e-0722dda8faaa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:08.913553 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:08.913384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shg5t\" (UniqueName: \"kubernetes.io/projected/6a60318f-3709-4820-890e-0722dda8faaa-kube-api-access-shg5t\") pod \"maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:08.913553 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:08.913438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a60318f-3709-4820-890e-0722dda8faaa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:08.913638 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:08.913584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a60318f-3709-4820-890e-0722dda8faaa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:08.947242 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:08.947210    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:09.014975 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:09.014939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a60318f-3709-4820-890e-0722dda8faaa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:09.015160 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:09.014983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shg5t\" (UniqueName: \"kubernetes.io/projected/6a60318f-3709-4820-890e-0722dda8faaa-kube-api-access-shg5t\") pod \"maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:09.015160 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:09.015025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a60318f-3709-4820-890e-0722dda8faaa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:09.015160 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:09.015134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a60318f-3709-4820-890e-0722dda8faaa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:09.015783 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:09.015757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a60318f-3709-4820-890e-0722dda8faaa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:09.017443 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:09.017414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a60318f-3709-4820-890e-0722dda8faaa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:09.017562 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:09.017541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a60318f-3709-4820-890e-0722dda8faaa-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:09.050510 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:09.050475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shg5t\" (UniqueName: \"kubernetes.io/projected/6a60318f-3709-4820-890e-0722dda8faaa-kube-api-access-shg5t\") pod \"maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:09.135627 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:09.135538    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:09.464414 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:09.464386    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"]
Sep 18 21:28:09.465449 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:28:09.465425    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a60318f_3709_4820_890e_0722dda8faaa.slice/crio-93ca1046419408f902473c9bc2a04b1d55821a71a5d340ba2c9f33b0814dee67 WatchSource:0}: Error finding container 93ca1046419408f902473c9bc2a04b1d55821a71a5d340ba2c9f33b0814dee67: Status 404 returned error can't find the container with id 93ca1046419408f902473c9bc2a04b1d55821a71a5d340ba2c9f33b0814dee67
Sep 18 21:28:09.472146 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:09.472107    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:09.866247 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:09.866210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m" event={"ID":"6a60318f-3709-4820-890e-0722dda8faaa","Type":"ContainerStarted","Data":"93ca1046419408f902473c9bc2a04b1d55821a71a5d340ba2c9f33b0814dee67"}
Sep 18 21:28:10.099256 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:10.099154    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:10.213503 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:10.213465    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:10.731877 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:10.731841    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:10.873511 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:10.873476    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m" event={"ID":"6a60318f-3709-4820-890e-0722dda8faaa","Type":"ContainerStarted","Data":"d7359a5aed830656815c4ac76b531c3a85b493a6588036ff0041feb5f5bdfc44"}
Sep 18 21:28:10.874037 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:10.873601    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:10.901698 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:10.901625    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m" podStartSLOduration=2.561663468 podStartE2EDuration="2.901604655s" totalImagesPullingTime="339.941187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:28:10.901485242 +0000 UTC m=+1328.508544801" watchObservedRunningTime="2026-09-18 21:28:10.901604655 +0000 UTC m=+1328.508664214"
Sep 18 21:28:11.570517 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:11.570483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:12.405302 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:12.405267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:13.551448 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:13.551415    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:14.243314 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:14.243279    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:15.718828 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:15.718789    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:16.884984 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:16.884954    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:28:36.552816 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:36.552720    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:36.787475 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:36.787438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:38.383357 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:38.383323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:39.254636 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:39.254597    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:40.135711 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:40.135607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:40.955018 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:40.954978    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:41.879952 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:41.879909    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:42.071423 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:42.071387    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:42.931147 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:42.931104    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:28:44.670737 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:28:44.670704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:06.254142 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.254105    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"]
Sep 18 21:29:06.257733 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.257714    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.264289 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.264252    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd993e\"/\"default-dockercfg-p7k9z\""
Sep 18 21:29:06.264459 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.264301    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd993e\"/\"kube-root-ca.crt\""
Sep 18 21:29:06.264459 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.264318    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd993e\"/\"openshift-service-ca.crt\""
Sep 18 21:29:06.264459 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.264370    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd993e\"/\"auth-test-model-fd993e-kserve-self-signed-certs\""
Sep 18 21:29:06.274168 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.274133    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"]
Sep 18 21:29:06.310864 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.310826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-tmp-dir\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.310864 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.310866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-model-cache\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.311082 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.310890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-home\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.311082 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.310949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc6rb\" (UniqueName: \"kubernetes.io/projected/a6575498-0945-47ad-8d21-f24b6fccbf55-kube-api-access-zc6rb\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.311082 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.310981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6575498-0945-47ad-8d21-f24b6fccbf55-tls-certs\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.311082 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.311015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-dshm\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.340515 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.340483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:06.412009 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.411965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-tmp-dir\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.412009 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.412013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-model-cache\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.412260 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.412035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-home\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.412260 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.412068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc6rb\" (UniqueName: \"kubernetes.io/projected/a6575498-0945-47ad-8d21-f24b6fccbf55-kube-api-access-zc6rb\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.412260 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.412097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6575498-0945-47ad-8d21-f24b6fccbf55-tls-certs\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.412260 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.412144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-dshm\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.412477 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.412450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-tmp-dir\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.412535 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.412484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-home\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.412592 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.412554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-model-cache\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.414625 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.414600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-dshm\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.414886 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.414870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6575498-0945-47ad-8d21-f24b6fccbf55-tls-certs\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.422751 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.422715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc6rb\" (UniqueName: \"kubernetes.io/projected/a6575498-0945-47ad-8d21-f24b6fccbf55-kube-api-access-zc6rb\") pod \"auth-test-model-fd993e-kserve-7d67dcd899-lv99k\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.567565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.567523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:06.808594 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.808557    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:06.872929 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:06.872845    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"]
Sep 18 21:29:06.876443 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:29:06.876412    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6575498_0945_47ad_8d21_f24b6fccbf55.slice/crio-47155ef8de98b06298853f147a644b2fea01ff92bdfca6514f39f00686a7b1e9 WatchSource:0}: Error finding container 47155ef8de98b06298853f147a644b2fea01ff92bdfca6514f39f00686a7b1e9: Status 404 returned error can't find the container with id 47155ef8de98b06298853f147a644b2fea01ff92bdfca6514f39f00686a7b1e9
Sep 18 21:29:07.196128 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:07.196022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k" event={"ID":"a6575498-0945-47ad-8d21-f24b6fccbf55","Type":"ContainerStarted","Data":"a4584ae841dfa562a44f35c0319268099e7f459a0b898c16a0f7489d73b7d295"}
Sep 18 21:29:07.196128 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:07.196071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k" event={"ID":"a6575498-0945-47ad-8d21-f24b6fccbf55","Type":"ContainerStarted","Data":"47155ef8de98b06298853f147a644b2fea01ff92bdfca6514f39f00686a7b1e9"}
Sep 18 21:29:07.223210 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:07.223153    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k" podStartSLOduration=1.223132104 podStartE2EDuration="1.223132104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:29:07.222117873 +0000 UTC m=+1384.829177431" watchObservedRunningTime="2026-09-18 21:29:07.223132104 +0000 UTC m=+1384.830191666"
Sep 18 21:29:08.191226 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:08.191194    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:09.189546 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:09.189516    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:10.326305 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:10.326259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:11.487215 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:11.487181    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:12.430023 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:12.429980    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:13.311915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:13.311873    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:14.533489 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:14.533453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:16.568349 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:16.568315    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:16.568770 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:16.568361    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:16.581199 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:16.581164    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:17.264912 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:17.264875    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:29:22.637948 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.637914    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"]
Sep 18 21:29:22.638419 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.638185    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5" podUID="abbddfa6-bb28-42e9-b377-a81fb1ced130" containerName="maas-api" containerID="cri-o://943c32d646b16285abcb722c1c435869d14f5c335d89a2b9ac759783a673c8e0" gracePeriod=30
Sep 18 21:29:22.669524 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.669488    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:22.761272 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.761236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:22.843211 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.843176    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:22.890404 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.890334    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:29:22.931534 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.931490    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:22.983296 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.983265    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbddfa6-bb28-42e9-b377-a81fb1ced130-maas-api-tls\") pod \"abbddfa6-bb28-42e9-b377-a81fb1ced130\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") "
Sep 18 21:29:22.983481 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.983312    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbddfa6-bb28-42e9-b377-a81fb1ced130-metrics-certs\") pod \"abbddfa6-bb28-42e9-b377-a81fb1ced130\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") "
Sep 18 21:29:22.983481 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.983337    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwrps\" (UniqueName: \"kubernetes.io/projected/abbddfa6-bb28-42e9-b377-a81fb1ced130-kube-api-access-fwrps\") pod \"abbddfa6-bb28-42e9-b377-a81fb1ced130\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") "
Sep 18 21:29:22.983481 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.983376    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbddfa6-bb28-42e9-b377-a81fb1ced130-openshift-service-ca\") pod \"abbddfa6-bb28-42e9-b377-a81fb1ced130\" (UID: \"abbddfa6-bb28-42e9-b377-a81fb1ced130\") "
Sep 18 21:29:22.983906 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.983874    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbddfa6-bb28-42e9-b377-a81fb1ced130-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "abbddfa6-bb28-42e9-b377-a81fb1ced130" (UID: "abbddfa6-bb28-42e9-b377-a81fb1ced130"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:29:22.985663 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.985615    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbddfa6-bb28-42e9-b377-a81fb1ced130-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "abbddfa6-bb28-42e9-b377-a81fb1ced130" (UID: "abbddfa6-bb28-42e9-b377-a81fb1ced130"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:29:22.985781 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.985716    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbddfa6-bb28-42e9-b377-a81fb1ced130-kube-api-access-fwrps" (OuterVolumeSpecName: "kube-api-access-fwrps") pod "abbddfa6-bb28-42e9-b377-a81fb1ced130" (UID: "abbddfa6-bb28-42e9-b377-a81fb1ced130"). InnerVolumeSpecName "kube-api-access-fwrps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:29:22.985952 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:22.985928    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbddfa6-bb28-42e9-b377-a81fb1ced130-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "abbddfa6-bb28-42e9-b377-a81fb1ced130" (UID: "abbddfa6-bb28-42e9-b377-a81fb1ced130"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:29:23.084845 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.084806    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbddfa6-bb28-42e9-b377-a81fb1ced130-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:29:23.084845 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.084839    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbddfa6-bb28-42e9-b377-a81fb1ced130-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:29:23.084845 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.084848    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbddfa6-bb28-42e9-b377-a81fb1ced130-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:29:23.085131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.084858    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwrps\" (UniqueName: \"kubernetes.io/projected/abbddfa6-bb28-42e9-b377-a81fb1ced130-kube-api-access-fwrps\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:29:23.285545 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.285510    2579 generic.go:356] "Generic (PLEG): container finished" podID="abbddfa6-bb28-42e9-b377-a81fb1ced130" containerID="943c32d646b16285abcb722c1c435869d14f5c335d89a2b9ac759783a673c8e0" exitCode=0
Sep 18 21:29:23.285756 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.285566    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"
Sep 18 21:29:23.285756 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.285595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5" event={"ID":"abbddfa6-bb28-42e9-b377-a81fb1ced130","Type":"ContainerDied","Data":"943c32d646b16285abcb722c1c435869d14f5c335d89a2b9ac759783a673c8e0"}
Sep 18 21:29:23.285756 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.285714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5" event={"ID":"abbddfa6-bb28-42e9-b377-a81fb1ced130","Type":"ContainerDied","Data":"5f83f7ff5d1e387e7e09bf9c76a3657c85c260d5759886d81436d72d55f67649"}
Sep 18 21:29:23.285756 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.285749    2579 scope.go:117] "RemoveContainer" containerID="943c32d646b16285abcb722c1c435869d14f5c335d89a2b9ac759783a673c8e0"
Sep 18 21:29:23.296298 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.296281    2579 scope.go:117] "RemoveContainer" containerID="943c32d646b16285abcb722c1c435869d14f5c335d89a2b9ac759783a673c8e0"
Sep 18 21:29:23.296550 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:29:23.296533    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943c32d646b16285abcb722c1c435869d14f5c335d89a2b9ac759783a673c8e0\": container with ID starting with 943c32d646b16285abcb722c1c435869d14f5c335d89a2b9ac759783a673c8e0 not found: ID does not exist" containerID="943c32d646b16285abcb722c1c435869d14f5c335d89a2b9ac759783a673c8e0"
Sep 18 21:29:23.296598 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.296556    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943c32d646b16285abcb722c1c435869d14f5c335d89a2b9ac759783a673c8e0"} err="failed to get container status \"943c32d646b16285abcb722c1c435869d14f5c335d89a2b9ac759783a673c8e0\": rpc error: code = NotFound desc = could not find container \"943c32d646b16285abcb722c1c435869d14f5c335d89a2b9ac759783a673c8e0\": container with ID starting with 943c32d646b16285abcb722c1c435869d14f5c335d89a2b9ac759783a673c8e0 not found: ID does not exist"
Sep 18 21:29:23.316051 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.316025    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"]
Sep 18 21:29:23.319788 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.319767    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67ccba3b-7c8957776f-vzpv5"]
Sep 18 21:29:23.348077 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:23.348050    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:24.261564 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:24.261524    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:24.923656 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:24.923616    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abbddfa6-bb28-42e9-b377-a81fb1ced130" path="/var/lib/kubelet/pods/abbddfa6-bb28-42e9-b377-a81fb1ced130/volumes"
Sep 18 21:29:25.244689 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:25.244578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:26.118890 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:26.118842    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:26.412198 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:26.412110    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:27.444533 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:27.444485    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:28.290373 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:28.290334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:29:31.203627 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:29:31.203585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:01.239627 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:01.239545    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:09.310875 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:09.310840    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:10.112819 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:10.112781    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:16.029398 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:16.029357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:16.129550 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:16.129509    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:16.301788 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:16.301760    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:17.628854 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:17.628819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:17.716231 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:17.716199    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:18.513385 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:18.513351    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:18.669863 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:18.669821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:19.934876 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:19.934819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:26.488248 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:26.488206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:26.574137 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:26.574096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:27.151539 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:27.151509    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:34.491519 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:34.491482    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:34.600241 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:34.600200    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:34.700054 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:34.700019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:35.044025 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:35.043995    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:36.410427 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:36.410393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:37.705812 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:37.705778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:38.843998 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:38.843947    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:39.813403 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:39.813371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:39.978041 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:39.978014    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"]
Sep 18 21:30:40.120230 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:40.120139    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:40.912435 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:40.912404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:42.069323 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:42.069288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:44.978839 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:44.978795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:45.079687 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:45.079635    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:45.210785 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:45.210753    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:45.450823 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:45.450793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:46.779603 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:46.779567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:50.933867 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:50.933831    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"]
Sep 18 21:30:50.934307 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:50.934058    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf" podUID="f76ac646-0715-400d-ad3d-fcb28f93ef9e" containerName="maas-api" containerID="cri-o://26063e6c5c0ea884b678880b723f6cea370b4b4c13aebe986af522cc1741b1e6" gracePeriod=30
Sep 18 21:30:50.965821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:50.965791    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:51.058358 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.058326    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:51.167349 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.167306    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:51.174114 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.174095    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:30:51.286959 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.286929    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28b5k\" (UniqueName: \"kubernetes.io/projected/f76ac646-0715-400d-ad3d-fcb28f93ef9e-kube-api-access-28b5k\") pod \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") "
Sep 18 21:30:51.287130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.287013    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ac646-0715-400d-ad3d-fcb28f93ef9e-openshift-service-ca\") pod \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") "
Sep 18 21:30:51.287130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.287110    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-metrics-certs\") pod \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") "
Sep 18 21:30:51.287250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.287137    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-maas-api-tls\") pod \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\" (UID: \"f76ac646-0715-400d-ad3d-fcb28f93ef9e\") "
Sep 18 21:30:51.287406 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.287381    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76ac646-0715-400d-ad3d-fcb28f93ef9e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f76ac646-0715-400d-ad3d-fcb28f93ef9e" (UID: "f76ac646-0715-400d-ad3d-fcb28f93ef9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:30:51.289085 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.289060    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76ac646-0715-400d-ad3d-fcb28f93ef9e-kube-api-access-28b5k" (OuterVolumeSpecName: "kube-api-access-28b5k") pod "f76ac646-0715-400d-ad3d-fcb28f93ef9e" (UID: "f76ac646-0715-400d-ad3d-fcb28f93ef9e"). InnerVolumeSpecName "kube-api-access-28b5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:30:51.289180 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.289133    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f76ac646-0715-400d-ad3d-fcb28f93ef9e" (UID: "f76ac646-0715-400d-ad3d-fcb28f93ef9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:30:51.289180 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.289158    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f76ac646-0715-400d-ad3d-fcb28f93ef9e" (UID: "f76ac646-0715-400d-ad3d-fcb28f93ef9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:30:51.388400 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.388364    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76ac646-0715-400d-ad3d-fcb28f93ef9e-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:30:51.388400 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.388394    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:30:51.388586 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.388407    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f76ac646-0715-400d-ad3d-fcb28f93ef9e-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:30:51.388586 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.388418    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28b5k\" (UniqueName: \"kubernetes.io/projected/f76ac646-0715-400d-ad3d-fcb28f93ef9e-kube-api-access-28b5k\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:30:51.428539 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.428511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:51.694655 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.694551    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:51.750754 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.750724    2579 generic.go:356] "Generic (PLEG): container finished" podID="f76ac646-0715-400d-ad3d-fcb28f93ef9e" containerID="26063e6c5c0ea884b678880b723f6cea370b4b4c13aebe986af522cc1741b1e6" exitCode=0
Sep 18 21:30:51.750913 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.750766    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"
Sep 18 21:30:51.750913 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.750775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf" event={"ID":"f76ac646-0715-400d-ad3d-fcb28f93ef9e","Type":"ContainerDied","Data":"26063e6c5c0ea884b678880b723f6cea370b4b4c13aebe986af522cc1741b1e6"}
Sep 18 21:30:51.750913 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.750814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf" event={"ID":"f76ac646-0715-400d-ad3d-fcb28f93ef9e","Type":"ContainerDied","Data":"20f82bd81839871c06bf935eea127fb181f41f9840e5faf08e482e1bfeeaa0d5"}
Sep 18 21:30:51.750913 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.750837    2579 scope.go:117] "RemoveContainer" containerID="26063e6c5c0ea884b678880b723f6cea370b4b4c13aebe986af522cc1741b1e6"
Sep 18 21:30:51.761007 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.760989    2579 scope.go:117] "RemoveContainer" containerID="26063e6c5c0ea884b678880b723f6cea370b4b4c13aebe986af522cc1741b1e6"
Sep 18 21:30:51.761343 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:30:51.761320    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26063e6c5c0ea884b678880b723f6cea370b4b4c13aebe986af522cc1741b1e6\": container with ID starting with 26063e6c5c0ea884b678880b723f6cea370b4b4c13aebe986af522cc1741b1e6 not found: ID does not exist" containerID="26063e6c5c0ea884b678880b723f6cea370b4b4c13aebe986af522cc1741b1e6"
Sep 18 21:30:51.761417 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.761353    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26063e6c5c0ea884b678880b723f6cea370b4b4c13aebe986af522cc1741b1e6"} err="failed to get container status \"26063e6c5c0ea884b678880b723f6cea370b4b4c13aebe986af522cc1741b1e6\": rpc error: code = NotFound desc = could not find container \"26063e6c5c0ea884b678880b723f6cea370b4b4c13aebe986af522cc1741b1e6\": container with ID starting with 26063e6c5c0ea884b678880b723f6cea370b4b4c13aebe986af522cc1741b1e6 not found: ID does not exist"
Sep 18 21:30:51.778622 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.778600    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"]
Sep 18 21:30:51.782284 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.782263    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6ad49b-5b86c8dbc5-2mkkf"]
Sep 18 21:30:51.886295 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:51.886271    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:52.923305 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:52.923268    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f76ac646-0715-400d-ad3d-fcb28f93ef9e" path="/var/lib/kubelet/pods/f76ac646-0715-400d-ad3d-fcb28f93ef9e/volumes"
Sep 18 21:30:53.265939 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:53.265857    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:54.441969 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:54.441939    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:55.454336 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:55.454298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:55.712612 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:55.712519    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz" podUID="3ce86f35-5681-4b5a-a9b5-fb706f8acb9c" containerName="main" containerID="cri-o://66dfc4f2ccad12e63d977a3072d2a8d94e6f83cbfca6e550a3d8108a5f0bff2f" gracePeriod=45
Sep 18 21:30:55.969454 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:55.969397    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:30:56.032533 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.032506    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-tls-certs\") pod \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") "
Sep 18 21:30:56.032707 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.032554    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-tmp-dir\") pod \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") "
Sep 18 21:30:56.032707 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.032592    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zx9c\" (UniqueName: \"kubernetes.io/projected/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-kube-api-access-7zx9c\") pod \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") "
Sep 18 21:30:56.032821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.032727    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-dshm\") pod \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") "
Sep 18 21:30:56.032880 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.032818    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-model-cache\") pod \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") "
Sep 18 21:30:56.032880 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.032859    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ce86f35-5681-4b5a-a9b5-fb706f8acb9c" (UID: "3ce86f35-5681-4b5a-a9b5-fb706f8acb9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:30:56.033099 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.032902    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-home\") pod \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\" (UID: \"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c\") "
Sep 18 21:30:56.033099 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.033062    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ce86f35-5681-4b5a-a9b5-fb706f8acb9c" (UID: "3ce86f35-5681-4b5a-a9b5-fb706f8acb9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:30:56.033235 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.033185    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-home" (OuterVolumeSpecName: "home") pod "3ce86f35-5681-4b5a-a9b5-fb706f8acb9c" (UID: "3ce86f35-5681-4b5a-a9b5-fb706f8acb9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:30:56.033404 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.033381    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-model-cache\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:30:56.033518 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.033408    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-home\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:30:56.033518 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.033422    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-tmp-dir\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:30:56.034928 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.034896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-kube-api-access-7zx9c" (OuterVolumeSpecName: "kube-api-access-7zx9c") pod "3ce86f35-5681-4b5a-a9b5-fb706f8acb9c" (UID: "3ce86f35-5681-4b5a-a9b5-fb706f8acb9c"). InnerVolumeSpecName "kube-api-access-7zx9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:30:56.035003 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.034939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-dshm" (OuterVolumeSpecName: "dshm") pod "3ce86f35-5681-4b5a-a9b5-fb706f8acb9c" (UID: "3ce86f35-5681-4b5a-a9b5-fb706f8acb9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:30:56.035157 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.035140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ce86f35-5681-4b5a-a9b5-fb706f8acb9c" (UID: "3ce86f35-5681-4b5a-a9b5-fb706f8acb9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:30:56.134335 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.134304    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-tls-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:30:56.134335 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.134329    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zx9c\" (UniqueName: \"kubernetes.io/projected/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-kube-api-access-7zx9c\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:30:56.134335 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.134339    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c-dshm\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:30:56.250413 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.250328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:56.779517 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.779482    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ce86f35-5681-4b5a-a9b5-fb706f8acb9c" containerID="66dfc4f2ccad12e63d977a3072d2a8d94e6f83cbfca6e550a3d8108a5f0bff2f" exitCode=0
Sep 18 21:30:56.780014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.779526    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"
Sep 18 21:30:56.780014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.779568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz" event={"ID":"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c","Type":"ContainerDied","Data":"66dfc4f2ccad12e63d977a3072d2a8d94e6f83cbfca6e550a3d8108a5f0bff2f"}
Sep 18 21:30:56.780014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.779615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz" event={"ID":"3ce86f35-5681-4b5a-a9b5-fb706f8acb9c","Type":"ContainerDied","Data":"bb35c514c7333fa997f0a49310cd09222531673e055ec725fd91b65e58706964"}
Sep 18 21:30:56.780014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.779636    2579 scope.go:117] "RemoveContainer" containerID="66dfc4f2ccad12e63d977a3072d2a8d94e6f83cbfca6e550a3d8108a5f0bff2f"
Sep 18 21:30:56.791110 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.791075    2579 scope.go:117] "RemoveContainer" containerID="66dfc4f2ccad12e63d977a3072d2a8d94e6f83cbfca6e550a3d8108a5f0bff2f"
Sep 18 21:30:56.791386 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:30:56.791365    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66dfc4f2ccad12e63d977a3072d2a8d94e6f83cbfca6e550a3d8108a5f0bff2f\": container with ID starting with 66dfc4f2ccad12e63d977a3072d2a8d94e6f83cbfca6e550a3d8108a5f0bff2f not found: ID does not exist" containerID="66dfc4f2ccad12e63d977a3072d2a8d94e6f83cbfca6e550a3d8108a5f0bff2f"
Sep 18 21:30:56.791461 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.791397    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66dfc4f2ccad12e63d977a3072d2a8d94e6f83cbfca6e550a3d8108a5f0bff2f"} err="failed to get container status \"66dfc4f2ccad12e63d977a3072d2a8d94e6f83cbfca6e550a3d8108a5f0bff2f\": rpc error: code = NotFound desc = could not find container \"66dfc4f2ccad12e63d977a3072d2a8d94e6f83cbfca6e550a3d8108a5f0bff2f\": container with ID starting with 66dfc4f2ccad12e63d977a3072d2a8d94e6f83cbfca6e550a3d8108a5f0bff2f not found: ID does not exist"
Sep 18 21:30:56.804313 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.804291    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"]
Sep 18 21:30:56.808924 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.808901    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6ad49b/distinct-w4-6ad49b-kserve-5f6976648f-mfvhz"]
Sep 18 21:30:56.921614 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:56.921580    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ce86f35-5681-4b5a-a9b5-fb706f8acb9c" path="/var/lib/kubelet/pods/3ce86f35-5681-4b5a-a9b5-fb706f8acb9c/volumes"
Sep 18 21:30:59.088047 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:59.088003    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:59.206477 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:59.206450    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:30:59.468871 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:30:59.468778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:02.928882 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:02.928857    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:31:02.932397 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:02.932371    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:31:02.934035 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:02.934012    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:31:02.937682 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:02.937663    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:31:06.377305 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:06.377271    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:13.312329 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:13.312294    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:13.420367 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:13.420335    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:13.676868 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:13.676781    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:14.005970 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:14.005879    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:14.132681 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:14.132628    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:14.367806 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:14.367773    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:17.212339 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:17.212305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:17.779227 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:17.779193    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:17.917500 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:17.917466    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:18.653916 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:18.653879    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:18.940096 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:18.939940    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:19.041178 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:19.041146    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:19.510250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:19.510216    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:20.346193 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:20.346154    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:20.747107 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:20.747020    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:21.475491 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:21.475456    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:22.293515 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.293481    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"]
Sep 18 21:31:22.293947 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.293932    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abbddfa6-bb28-42e9-b377-a81fb1ced130" containerName="maas-api"
Sep 18 21:31:22.294001 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.293949    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="abbddfa6-bb28-42e9-b377-a81fb1ced130" containerName="maas-api"
Sep 18 21:31:22.294001 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.293976    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f76ac646-0715-400d-ad3d-fcb28f93ef9e" containerName="maas-api"
Sep 18 21:31:22.294001 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.293982    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f76ac646-0715-400d-ad3d-fcb28f93ef9e" containerName="maas-api"
Sep 18 21:31:22.294001 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.293988    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce86f35-5681-4b5a-a9b5-fb706f8acb9c" containerName="main"
Sep 18 21:31:22.294001 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.293993    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce86f35-5681-4b5a-a9b5-fb706f8acb9c" containerName="main"
Sep 18 21:31:22.294165 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.294070    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="abbddfa6-bb28-42e9-b377-a81fb1ced130" containerName="maas-api"
Sep 18 21:31:22.294165 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.294079    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce86f35-5681-4b5a-a9b5-fb706f8acb9c" containerName="main"
Sep 18 21:31:22.294165 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.294089    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f76ac646-0715-400d-ad3d-fcb28f93ef9e" containerName="maas-api"
Sep 18 21:31:22.297127 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.297108    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:22.303174 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.303151    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8424a6-serving-cert\""
Sep 18 21:31:22.321457 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.321431    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"]
Sep 18 21:31:22.352507 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.352479    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:22.370884 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.370857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-metrics-certs\") pod \"maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:22.371005 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.370914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-maas-api-tls\") pod \"maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:22.371063 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.371012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a5ce5b-224f-4cef-b1b4-feae48a373c0-openshift-service-ca\") pod \"maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:22.371114 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.371087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvzvz\" (UniqueName: \"kubernetes.io/projected/24a5ce5b-224f-4cef-b1b4-feae48a373c0-kube-api-access-tvzvz\") pod \"maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:22.454197 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.454163    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:22.471549 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.471517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvzvz\" (UniqueName: \"kubernetes.io/projected/24a5ce5b-224f-4cef-b1b4-feae48a373c0-kube-api-access-tvzvz\") pod \"maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:22.471698 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.471568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-metrics-certs\") pod \"maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:22.471698 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.471669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-maas-api-tls\") pod \"maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:22.471821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.471725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a5ce5b-224f-4cef-b1b4-feae48a373c0-openshift-service-ca\") pod \"maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:22.471821 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:31:22.471773    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-serving-cert: secret "maas-api-e2e-api-a-8424a6-serving-cert" not found
Sep 18 21:31:22.471908 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:31:22.471865    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-maas-api-tls podName:24a5ce5b-224f-4cef-b1b4-feae48a373c0 nodeName:}" failed. No retries permitted until 2026-09-18 21:31:22.971835611 +0000 UTC m=+1520.578895163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-maas-api-tls") pod "maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4" (UID: "24a5ce5b-224f-4cef-b1b4-feae48a373c0") : secret "maas-api-e2e-api-a-8424a6-serving-cert" not found
Sep 18 21:31:22.472339 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.472322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a5ce5b-224f-4cef-b1b4-feae48a373c0-openshift-service-ca\") pod \"maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:22.474231 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.474213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-metrics-certs\") pod \"maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:22.481159 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.481138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvzvz\" (UniqueName: \"kubernetes.io/projected/24a5ce5b-224f-4cef-b1b4-feae48a373c0-kube-api-access-tvzvz\") pod \"maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:22.546451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.546394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:22.976200 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.976118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-maas-api-tls\") pod \"maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:22.978466 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:22.978443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-maas-api-tls\") pod \"maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:23.021311 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:23.021278    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:23.114451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:23.114404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:23.207158 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:23.207130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:23.500360 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:23.500328    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"]
Sep 18 21:31:23.504323 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:31:23.504284    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a5ce5b_224f_4cef_b1b4_feae48a373c0.slice/crio-a7d8ec79ae75ddc7fb07947d0a67ca22e80bc77d97d925f45eb093b49fffd305 WatchSource:0}: Error finding container a7d8ec79ae75ddc7fb07947d0a67ca22e80bc77d97d925f45eb093b49fffd305: Status 404 returned error can't find the container with id a7d8ec79ae75ddc7fb07947d0a67ca22e80bc77d97d925f45eb093b49fffd305
Sep 18 21:31:23.921072 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:23.921044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4" event={"ID":"24a5ce5b-224f-4cef-b1b4-feae48a373c0","Type":"ContainerStarted","Data":"a7d8ec79ae75ddc7fb07947d0a67ca22e80bc77d97d925f45eb093b49fffd305"}
Sep 18 21:31:24.927046 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:24.927006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4" event={"ID":"24a5ce5b-224f-4cef-b1b4-feae48a373c0","Type":"ContainerStarted","Data":"418240a14e22e118c623ab1e94ba7ecef3fa0cf576285ed8ea89b233ca09b23f"}
Sep 18 21:31:24.927468 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:24.927129    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:24.949686 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:24.949630    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4" podStartSLOduration=2.540594871 podStartE2EDuration="2.949613663s" totalImagesPullingTime="409.018792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:31:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:31:24.947097745 +0000 UTC m=+1522.554157303" watchObservedRunningTime="2026-09-18 21:31:24.949613663 +0000 UTC m=+1522.556673222"
Sep 18 21:31:30.936620 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:30.936587    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:31:39.174611 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:39.174579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:39.632258 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:39.632209    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:44.990310 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:44.990273    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:45.632633 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:45.632596    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:46.424529 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:46.424495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:46.680896 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:46.680803    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:47.642212 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:47.642174    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:47.800288 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:47.800249    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:48.254155 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:48.254120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:48.477339 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:48.477307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:48.619509 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:48.619469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:48.999742 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:48.999629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:49.405156 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:49.405121    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:50.525665 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:50.525607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:51.523972 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:51.523937    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:51.997980 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:51.997943    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:52.137745 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:52.137711    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:52.542723 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:52.542692    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:53.447388 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:53.447352    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:54.303463 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:54.303434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:54.727531 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:54.727435    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:55.118591 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:55.118554    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"]
Sep 18 21:31:55.132713 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:55.132685    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:55.496587 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:55.496504    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:56.348181 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:56.348131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:57.125499 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.125466    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"]
Sep 18 21:31:57.129028 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.129010    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:57.133050 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.133025    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9e75fb-serving-cert\""
Sep 18 21:31:57.143975 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.143951    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"]
Sep 18 21:31:57.175354 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.175309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:57.278833 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.278800    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:57.308029 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.307998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-maas-api-tls\") pod \"maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:57.308149 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.308062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-metrics-certs\") pod \"maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:57.308149 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.308136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97b7k\" (UniqueName: \"kubernetes.io/projected/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-kube-api-access-97b7k\") pod \"maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:57.308220 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.308182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-openshift-service-ca\") pod \"maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:57.409550 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.409469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-metrics-certs\") pod \"maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:57.409550 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.409517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97b7k\" (UniqueName: \"kubernetes.io/projected/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-kube-api-access-97b7k\") pod \"maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:57.409966 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.409638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-openshift-service-ca\") pod \"maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:57.409966 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.409769    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-maas-api-tls\") pod \"maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:57.409966 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:31:57.409874    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-serving-cert: secret "maas-api-e2e-api-b-9e75fb-serving-cert" not found
Sep 18 21:31:57.409966 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:31:57.409936    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-maas-api-tls podName:ef5903bc-60a2-45bb-a8a5-1c48b4b5613d nodeName:}" failed. No retries permitted until 2026-09-18 21:31:57.909921287 +0000 UTC m=+1555.516980828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-maas-api-tls") pod "maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x" (UID: "ef5903bc-60a2-45bb-a8a5-1c48b4b5613d") : secret "maas-api-e2e-api-b-9e75fb-serving-cert" not found
Sep 18 21:31:57.410302 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.410285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-openshift-service-ca\") pod \"maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:57.412049 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.412028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-metrics-certs\") pod \"maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:57.422974 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.422955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97b7k\" (UniqueName: \"kubernetes.io/projected/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-kube-api-access-97b7k\") pod \"maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:57.749951 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.749852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:57.911030 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.910998    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:31:57.916299 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.916268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-maas-api-tls\") pod \"maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:57.918717 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:57.918687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-maas-api-tls\") pod \"maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:58.041036 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:58.041003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:58.321674 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:58.321629    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"]
Sep 18 21:31:58.323338 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:31:58.323314    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5903bc_60a2_45bb_a8a5_1c48b4b5613d.slice/crio-471cea458873630b47d89ad2b01be73f0c0f50184c2577d051a0cb63e6cf070f WatchSource:0}: Error finding container 471cea458873630b47d89ad2b01be73f0c0f50184c2577d051a0cb63e6cf070f: Status 404 returned error can't find the container with id 471cea458873630b47d89ad2b01be73f0c0f50184c2577d051a0cb63e6cf070f
Sep 18 21:31:59.117022 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:59.116980    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x" event={"ID":"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d","Type":"ContainerStarted","Data":"4b73b39b6ebb1c1f1c0d5e4223458402a76414c16718cdf5c0cda8e9cb5e038e"}
Sep 18 21:31:59.117405 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:59.117049    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:31:59.117405 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:59.117070    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x" event={"ID":"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d","Type":"ContainerStarted","Data":"471cea458873630b47d89ad2b01be73f0c0f50184c2577d051a0cb63e6cf070f"}
Sep 18 21:31:59.141325 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:31:59.141263    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x" podStartSLOduration=1.779849529 podStartE2EDuration="2.141247055s" totalImagesPullingTime="361.397526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:31:59.139971983 +0000 UTC m=+1556.747031545" watchObservedRunningTime="2026-09-18 21:31:59.141247055 +0000 UTC m=+1556.748306612"
Sep 18 21:32:05.128503 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:05.128468    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:32:10.138264 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.138226    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k" podUID="a6575498-0945-47ad-8d21-f24b6fccbf55" containerName="main" containerID="cri-o://a4584ae841dfa562a44f35c0319268099e7f459a0b898c16a0f7489d73b7d295" gracePeriod=45
Sep 18 21:32:10.380821 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.380798    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:32:10.427577 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.427513    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-model-cache\") pod \"a6575498-0945-47ad-8d21-f24b6fccbf55\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") "
Sep 18 21:32:10.427577 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.427562    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-dshm\") pod \"a6575498-0945-47ad-8d21-f24b6fccbf55\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") "
Sep 18 21:32:10.427739 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.427618    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-tmp-dir\") pod \"a6575498-0945-47ad-8d21-f24b6fccbf55\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") "
Sep 18 21:32:10.427739 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.427666    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc6rb\" (UniqueName: \"kubernetes.io/projected/a6575498-0945-47ad-8d21-f24b6fccbf55-kube-api-access-zc6rb\") pod \"a6575498-0945-47ad-8d21-f24b6fccbf55\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") "
Sep 18 21:32:10.427739 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.427713    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-home\") pod \"a6575498-0945-47ad-8d21-f24b6fccbf55\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") "
Sep 18 21:32:10.427739 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.427738    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6575498-0945-47ad-8d21-f24b6fccbf55-tls-certs\") pod \"a6575498-0945-47ad-8d21-f24b6fccbf55\" (UID: \"a6575498-0945-47ad-8d21-f24b6fccbf55\") "
Sep 18 21:32:10.427946 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.427795    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6575498-0945-47ad-8d21-f24b6fccbf55" (UID: "a6575498-0945-47ad-8d21-f24b6fccbf55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:32:10.427946 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.427886    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6575498-0945-47ad-8d21-f24b6fccbf55" (UID: "a6575498-0945-47ad-8d21-f24b6fccbf55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:32:10.428058 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.428003    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-home" (OuterVolumeSpecName: "home") pod "a6575498-0945-47ad-8d21-f24b6fccbf55" (UID: "a6575498-0945-47ad-8d21-f24b6fccbf55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:32:10.428132 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.428113    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-tmp-dir\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:32:10.428204 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.428137    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-home\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:32:10.428204 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.428151    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-model-cache\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:32:10.429979 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.429953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6575498-0945-47ad-8d21-f24b6fccbf55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6575498-0945-47ad-8d21-f24b6fccbf55" (UID: "a6575498-0945-47ad-8d21-f24b6fccbf55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:32:10.430244 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.430216    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-dshm" (OuterVolumeSpecName: "dshm") pod "a6575498-0945-47ad-8d21-f24b6fccbf55" (UID: "a6575498-0945-47ad-8d21-f24b6fccbf55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:32:10.430244 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.430228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6575498-0945-47ad-8d21-f24b6fccbf55-kube-api-access-zc6rb" (OuterVolumeSpecName: "kube-api-access-zc6rb") pod "a6575498-0945-47ad-8d21-f24b6fccbf55" (UID: "a6575498-0945-47ad-8d21-f24b6fccbf55"). InnerVolumeSpecName "kube-api-access-zc6rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:32:10.529130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.529106    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6575498-0945-47ad-8d21-f24b6fccbf55-dshm\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:32:10.529130 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.529128    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc6rb\" (UniqueName: \"kubernetes.io/projected/a6575498-0945-47ad-8d21-f24b6fccbf55-kube-api-access-zc6rb\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:32:10.529269 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:10.529139    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6575498-0945-47ad-8d21-f24b6fccbf55-tls-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:32:11.185497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:11.185416    2579 generic.go:356] "Generic (PLEG): container finished" podID="a6575498-0945-47ad-8d21-f24b6fccbf55" containerID="a4584ae841dfa562a44f35c0319268099e7f459a0b898c16a0f7489d73b7d295" exitCode=0
Sep 18 21:32:11.185497 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:11.185455    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"
Sep 18 21:32:11.185978 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:11.185498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k" event={"ID":"a6575498-0945-47ad-8d21-f24b6fccbf55","Type":"ContainerDied","Data":"a4584ae841dfa562a44f35c0319268099e7f459a0b898c16a0f7489d73b7d295"}
Sep 18 21:32:11.185978 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:11.185541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k" event={"ID":"a6575498-0945-47ad-8d21-f24b6fccbf55","Type":"ContainerDied","Data":"47155ef8de98b06298853f147a644b2fea01ff92bdfca6514f39f00686a7b1e9"}
Sep 18 21:32:11.185978 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:11.185564    2579 scope.go:117] "RemoveContainer" containerID="a4584ae841dfa562a44f35c0319268099e7f459a0b898c16a0f7489d73b7d295"
Sep 18 21:32:11.194735 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:11.194719    2579 scope.go:117] "RemoveContainer" containerID="a4584ae841dfa562a44f35c0319268099e7f459a0b898c16a0f7489d73b7d295"
Sep 18 21:32:11.195006 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:32:11.194988    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4584ae841dfa562a44f35c0319268099e7f459a0b898c16a0f7489d73b7d295\": container with ID starting with a4584ae841dfa562a44f35c0319268099e7f459a0b898c16a0f7489d73b7d295 not found: ID does not exist" containerID="a4584ae841dfa562a44f35c0319268099e7f459a0b898c16a0f7489d73b7d295"
Sep 18 21:32:11.195061 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:11.195016    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4584ae841dfa562a44f35c0319268099e7f459a0b898c16a0f7489d73b7d295"} err="failed to get container status \"a4584ae841dfa562a44f35c0319268099e7f459a0b898c16a0f7489d73b7d295\": rpc error: code = NotFound desc = could not find container \"a4584ae841dfa562a44f35c0319268099e7f459a0b898c16a0f7489d73b7d295\": container with ID starting with a4584ae841dfa562a44f35c0319268099e7f459a0b898c16a0f7489d73b7d295 not found: ID does not exist"
Sep 18 21:32:11.203708 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:11.203678    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"]
Sep 18 21:32:11.208717 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:11.208696    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/auth-test-model-fd993e-kserve-7d67dcd899-lv99k"]
Sep 18 21:32:12.923941 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:12.923907    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6575498-0945-47ad-8d21-f24b6fccbf55" path="/var/lib/kubelet/pods/a6575498-0945-47ad-8d21-f24b6fccbf55/volumes"
Sep 18 21:32:16.799141 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:16.799106    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:19.969995 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:19.969959    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:20.852893 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:20.852863    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:21.845697 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:21.845663    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:22.287239 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:22.287207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:22.938026 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:22.937995    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:23.823479 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:23.823437    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:24.972548 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:24.972508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:25.938110 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:25.938079    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:27.102256 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:27.102219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:28.221177 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:28.221141    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:28.987222 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:28.987192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:30.207506 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:30.207473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:31.615539 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:31.615506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:32.811976 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:32.811941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:35.745286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:35.745245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:41.412713 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.412271    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"]
Sep 18 21:32:41.412713 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.412550    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4" podUID="24a5ce5b-224f-4cef-b1b4-feae48a373c0" containerName="maas-api" containerID="cri-o://418240a14e22e118c623ab1e94ba7ecef3fa0cf576285ed8ea89b233ca09b23f" gracePeriod=30
Sep 18 21:32:41.442358 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.442324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:41.521694 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.521659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:41.623634 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.623600    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:41.668872 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.668805    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:32:41.698687 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.698655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:41.733974 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.733941    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvzvz\" (UniqueName: \"kubernetes.io/projected/24a5ce5b-224f-4cef-b1b4-feae48a373c0-kube-api-access-tvzvz\") pod \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") "
Sep 18 21:32:41.734152 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.734132    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-metrics-certs\") pod \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") "
Sep 18 21:32:41.734208 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.734168    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a5ce5b-224f-4cef-b1b4-feae48a373c0-openshift-service-ca\") pod \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") "
Sep 18 21:32:41.734270 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.734206    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-maas-api-tls\") pod \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\" (UID: \"24a5ce5b-224f-4cef-b1b4-feae48a373c0\") "
Sep 18 21:32:41.734670 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.734610    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a5ce5b-224f-4cef-b1b4-feae48a373c0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24a5ce5b-224f-4cef-b1b4-feae48a373c0" (UID: "24a5ce5b-224f-4cef-b1b4-feae48a373c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:32:41.736182 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.736157    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a5ce5b-224f-4cef-b1b4-feae48a373c0-kube-api-access-tvzvz" (OuterVolumeSpecName: "kube-api-access-tvzvz") pod "24a5ce5b-224f-4cef-b1b4-feae48a373c0" (UID: "24a5ce5b-224f-4cef-b1b4-feae48a373c0"). InnerVolumeSpecName "kube-api-access-tvzvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:32:41.736391 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.736371    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24a5ce5b-224f-4cef-b1b4-feae48a373c0" (UID: "24a5ce5b-224f-4cef-b1b4-feae48a373c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:32:41.736532 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.736516    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "24a5ce5b-224f-4cef-b1b4-feae48a373c0" (UID: "24a5ce5b-224f-4cef-b1b4-feae48a373c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:32:41.835910 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.835875    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvzvz\" (UniqueName: \"kubernetes.io/projected/24a5ce5b-224f-4cef-b1b4-feae48a373c0-kube-api-access-tvzvz\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:32:41.835910 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.835904    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:32:41.835910 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.835913    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a5ce5b-224f-4cef-b1b4-feae48a373c0-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:32:41.836201 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:41.835923    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24a5ce5b-224f-4cef-b1b4-feae48a373c0-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:32:42.045288 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:42.045259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:42.146137 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:42.146110    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:42.356480 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:42.356402    2579 generic.go:356] "Generic (PLEG): container finished" podID="24a5ce5b-224f-4cef-b1b4-feae48a373c0" containerID="418240a14e22e118c623ab1e94ba7ecef3fa0cf576285ed8ea89b233ca09b23f" exitCode=0
Sep 18 21:32:42.356480 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:42.356452    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"
Sep 18 21:32:42.356665 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:42.356493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4" event={"ID":"24a5ce5b-224f-4cef-b1b4-feae48a373c0","Type":"ContainerDied","Data":"418240a14e22e118c623ab1e94ba7ecef3fa0cf576285ed8ea89b233ca09b23f"}
Sep 18 21:32:42.356665 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:42.356540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4" event={"ID":"24a5ce5b-224f-4cef-b1b4-feae48a373c0","Type":"ContainerDied","Data":"a7d8ec79ae75ddc7fb07947d0a67ca22e80bc77d97d925f45eb093b49fffd305"}
Sep 18 21:32:42.356665 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:42.356564    2579 scope.go:117] "RemoveContainer" containerID="418240a14e22e118c623ab1e94ba7ecef3fa0cf576285ed8ea89b233ca09b23f"
Sep 18 21:32:42.372527 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:42.372507    2579 scope.go:117] "RemoveContainer" containerID="418240a14e22e118c623ab1e94ba7ecef3fa0cf576285ed8ea89b233ca09b23f"
Sep 18 21:32:42.372807 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:32:42.372787    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418240a14e22e118c623ab1e94ba7ecef3fa0cf576285ed8ea89b233ca09b23f\": container with ID starting with 418240a14e22e118c623ab1e94ba7ecef3fa0cf576285ed8ea89b233ca09b23f not found: ID does not exist" containerID="418240a14e22e118c623ab1e94ba7ecef3fa0cf576285ed8ea89b233ca09b23f"
Sep 18 21:32:42.372886 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:42.372815    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418240a14e22e118c623ab1e94ba7ecef3fa0cf576285ed8ea89b233ca09b23f"} err="failed to get container status \"418240a14e22e118c623ab1e94ba7ecef3fa0cf576285ed8ea89b233ca09b23f\": rpc error: code = NotFound desc = could not find container \"418240a14e22e118c623ab1e94ba7ecef3fa0cf576285ed8ea89b233ca09b23f\": container with ID starting with 418240a14e22e118c623ab1e94ba7ecef3fa0cf576285ed8ea89b233ca09b23f not found: ID does not exist"
Sep 18 21:32:42.395975 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:42.395949    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"]
Sep 18 21:32:42.399354 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:42.399331    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8424a6-cb5f8f7d-x8fm4"]
Sep 18 21:32:42.922125 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:42.922092    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24a5ce5b-224f-4cef-b1b4-feae48a373c0" path="/var/lib/kubelet/pods/24a5ce5b-224f-4cef-b1b4-feae48a373c0/volumes"
Sep 18 21:32:43.011504 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:43.011477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:43.807836 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:43.807799    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:44.726524 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:44.726487    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:45.637266 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:45.637220    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:46.351316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:46.351283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:47.157387 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.157354    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"]
Sep 18 21:32:47.157810 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.157796    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6575498-0945-47ad-8d21-f24b6fccbf55" containerName="main"
Sep 18 21:32:47.157872 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.157812    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6575498-0945-47ad-8d21-f24b6fccbf55" containerName="main"
Sep 18 21:32:47.157872 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.157835    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a5ce5b-224f-4cef-b1b4-feae48a373c0" containerName="maas-api"
Sep 18 21:32:47.157872 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.157842    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a5ce5b-224f-4cef-b1b4-feae48a373c0" containerName="maas-api"
Sep 18 21:32:47.158028 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.157911    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6575498-0945-47ad-8d21-f24b6fccbf55" containerName="main"
Sep 18 21:32:47.158028 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.157921    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a5ce5b-224f-4cef-b1b4-feae48a373c0" containerName="maas-api"
Sep 18 21:32:47.161911 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.161880    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.169015 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.168997    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd993e\"/\"kube-root-ca.crt\""
Sep 18 21:32:47.169941 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.169922    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd993e\"/\"default-dockercfg-p7k9z\""
Sep 18 21:32:47.170050 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.169990    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd993e\"/\"sub-test-model-fd993e-kserve-self-signed-certs\""
Sep 18 21:32:47.170180 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.170164    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd993e\"/\"openshift-service-ca.crt\""
Sep 18 21:32:47.182464 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.182441    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"]
Sep 18 21:32:47.235244 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.235213    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:47.283984 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.283958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-dshm\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.284123 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.283993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw4tv\" (UniqueName: \"kubernetes.io/projected/f10c3a3c-6955-482b-a20d-010c5c343ea3-kube-api-access-jw4tv\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.284123 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.284092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10c3a3c-6955-482b-a20d-010c5c343ea3-tls-certs\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.284198 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.284148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-tmp-dir\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.284198 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.284173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-model-cache\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.284265 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.284204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-home\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.385007 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.384982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw4tv\" (UniqueName: \"kubernetes.io/projected/f10c3a3c-6955-482b-a20d-010c5c343ea3-kube-api-access-jw4tv\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.385334 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.385051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10c3a3c-6955-482b-a20d-010c5c343ea3-tls-certs\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.385334 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.385098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-tmp-dir\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.385334 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.385120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-model-cache\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.385334 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.385151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-home\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.385334 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.385201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-dshm\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.385568 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.385551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-model-cache\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.385626 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.385568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-home\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.385626 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.385611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-tmp-dir\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.387251 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.387229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-dshm\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.388075 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.388046    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10c3a3c-6955-482b-a20d-010c5c343ea3-tls-certs\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.398891 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.398852    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw4tv\" (UniqueName: \"kubernetes.io/projected/f10c3a3c-6955-482b-a20d-010c5c343ea3-kube-api-access-jw4tv\") pod \"sub-test-model-fd993e-kserve-7f87f8b855-dn7rb\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.471341 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.471285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:47.752178 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:47.752152    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"]
Sep 18 21:32:47.754746 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:32:47.754710    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf10c3a3c_6955_482b_a20d_010c5c343ea3.slice/crio-ee15b27f2f0cd9b70aee15eb072e251d4509d46c0b5ad66d425949ca10edfb3a WatchSource:0}: Error finding container ee15b27f2f0cd9b70aee15eb072e251d4509d46c0b5ad66d425949ca10edfb3a: Status 404 returned error can't find the container with id ee15b27f2f0cd9b70aee15eb072e251d4509d46c0b5ad66d425949ca10edfb3a
Sep 18 21:32:48.389884 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:48.389851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb" event={"ID":"f10c3a3c-6955-482b-a20d-010c5c343ea3","Type":"ContainerStarted","Data":"214b1b169e52ed82a9fdd8d26fc0392fafd1969c33fee25b74aa18bbea029c16"}
Sep 18 21:32:48.390242 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:48.389895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb" event={"ID":"f10c3a3c-6955-482b-a20d-010c5c343ea3","Type":"ContainerStarted","Data":"ee15b27f2f0cd9b70aee15eb072e251d4509d46c0b5ad66d425949ca10edfb3a"}
Sep 18 21:32:48.416095 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:48.416031    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb" podStartSLOduration=1.416014148 podStartE2EDuration="1.416014148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:32:48.413654034 +0000 UTC m=+1606.020713585" watchObservedRunningTime="2026-09-18 21:32:48.416014148 +0000 UTC m=+1606.023073706"
Sep 18 21:32:49.087855 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:49.087820    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:49.162217 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:49.162186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:49.352729 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:49.352607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:49.978933 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:49.978893    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:50.837834 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:50.837803    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:51.599330 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:51.599297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:52.654613 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:52.654580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:53.484904 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:53.484869    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:54.743763 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:54.743726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:55.532055 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:55.531975    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:32:57.472131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:57.472099    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:57.472131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:57.472142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:57.484695 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:57.484668    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:32:58.453412 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:32:58.453383    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:33:01.724022 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:01.723981    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:05.221873 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.221822    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"]
Sep 18 21:33:05.222294 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.222044    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x" podUID="ef5903bc-60a2-45bb-a8a5-1c48b4b5613d" containerName="maas-api" containerID="cri-o://4b73b39b6ebb1c1f1c0d5e4223458402a76414c16718cdf5c0cda8e9cb5e038e" gracePeriod=30
Sep 18 21:33:05.262137 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.262014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:05.359479 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.359440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:05.456847 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.456808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:05.470523 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.470498    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:33:05.480905 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.480850    2579 generic.go:356] "Generic (PLEG): container finished" podID="ef5903bc-60a2-45bb-a8a5-1c48b4b5613d" containerID="4b73b39b6ebb1c1f1c0d5e4223458402a76414c16718cdf5c0cda8e9cb5e038e" exitCode=0
Sep 18 21:33:05.480905 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.480884    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"
Sep 18 21:33:05.481044 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.480935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x" event={"ID":"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d","Type":"ContainerDied","Data":"4b73b39b6ebb1c1f1c0d5e4223458402a76414c16718cdf5c0cda8e9cb5e038e"}
Sep 18 21:33:05.481044 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.480992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x" event={"ID":"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d","Type":"ContainerDied","Data":"471cea458873630b47d89ad2b01be73f0c0f50184c2577d051a0cb63e6cf070f"}
Sep 18 21:33:05.481044 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.481018    2579 scope.go:117] "RemoveContainer" containerID="4b73b39b6ebb1c1f1c0d5e4223458402a76414c16718cdf5c0cda8e9cb5e038e"
Sep 18 21:33:05.489443 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.489426    2579 scope.go:117] "RemoveContainer" containerID="4b73b39b6ebb1c1f1c0d5e4223458402a76414c16718cdf5c0cda8e9cb5e038e"
Sep 18 21:33:05.489774 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:33:05.489756    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b73b39b6ebb1c1f1c0d5e4223458402a76414c16718cdf5c0cda8e9cb5e038e\": container with ID starting with 4b73b39b6ebb1c1f1c0d5e4223458402a76414c16718cdf5c0cda8e9cb5e038e not found: ID does not exist" containerID="4b73b39b6ebb1c1f1c0d5e4223458402a76414c16718cdf5c0cda8e9cb5e038e"
Sep 18 21:33:05.489836 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.489782    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b73b39b6ebb1c1f1c0d5e4223458402a76414c16718cdf5c0cda8e9cb5e038e"} err="failed to get container status \"4b73b39b6ebb1c1f1c0d5e4223458402a76414c16718cdf5c0cda8e9cb5e038e\": rpc error: code = NotFound desc = could not find container \"4b73b39b6ebb1c1f1c0d5e4223458402a76414c16718cdf5c0cda8e9cb5e038e\": container with ID starting with 4b73b39b6ebb1c1f1c0d5e4223458402a76414c16718cdf5c0cda8e9cb5e038e not found: ID does not exist"
Sep 18 21:33:05.543136 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.543103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:05.550319 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.550300    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-openshift-service-ca\") pod \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") "
Sep 18 21:33:05.550409 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.550368    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97b7k\" (UniqueName: \"kubernetes.io/projected/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-kube-api-access-97b7k\") pod \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") "
Sep 18 21:33:05.550459 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.550447    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-metrics-certs\") pod \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") "
Sep 18 21:33:05.550505 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.550494    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-maas-api-tls\") pod \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\" (UID: \"ef5903bc-60a2-45bb-a8a5-1c48b4b5613d\") "
Sep 18 21:33:05.550704 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.550678    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef5903bc-60a2-45bb-a8a5-1c48b4b5613d" (UID: "ef5903bc-60a2-45bb-a8a5-1c48b4b5613d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:33:05.550788 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.550775    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:05.552479 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.552451    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-kube-api-access-97b7k" (OuterVolumeSpecName: "kube-api-access-97b7k") pod "ef5903bc-60a2-45bb-a8a5-1c48b4b5613d" (UID: "ef5903bc-60a2-45bb-a8a5-1c48b4b5613d"). InnerVolumeSpecName "kube-api-access-97b7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:33:05.552580 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.552506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ef5903bc-60a2-45bb-a8a5-1c48b4b5613d" (UID: "ef5903bc-60a2-45bb-a8a5-1c48b4b5613d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:33:05.552580 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.552568    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef5903bc-60a2-45bb-a8a5-1c48b4b5613d" (UID: "ef5903bc-60a2-45bb-a8a5-1c48b4b5613d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:33:05.651943 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.651914    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:05.651943 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.651939    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:05.651943 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.651948    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97b7k\" (UniqueName: \"kubernetes.io/projected/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d-kube-api-access-97b7k\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:05.804403 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.804373    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"]
Sep 18 21:33:05.806327 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.806304    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9e75fb-768fb66498-g5s6x"]
Sep 18 21:33:05.901858 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:05.901821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:06.002452 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:06.002422    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:06.643770 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:06.643731    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:06.921937 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:06.921865    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef5903bc-60a2-45bb-a8a5-1c48b4b5613d" path="/var/lib/kubelet/pods/ef5903bc-60a2-45bb-a8a5-1c48b4b5613d/volumes"
Sep 18 21:33:07.642669 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:07.642616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:08.389548 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:08.389511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:09.297150 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:09.297117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:12.210604 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:12.210567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:12.367484 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:12.367453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:12.651024 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:12.650988    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:13.212184 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:13.212152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:14.121197 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:14.121161    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:14.916958 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:14.916926    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:15.747999 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:15.747966    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:16.446955 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:16.446919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:17.352624 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:17.352591    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:23.830063 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:23.830027    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"]
Sep 18 21:33:23.830538 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:23.830286    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh" podUID="b9ad9f28-88ec-450e-95fa-0e812c2de534" containerName="maas-api" containerID="cri-o://abe8c51ef42f1096eded7dbe412bdc969815547f4f2f2cc97b67631110cef8c4" gracePeriod=30
Sep 18 21:33:23.866234 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:23.866207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:23.980434 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:23.980385    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:24.081051 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.080988    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:33:24.100297 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.100270    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:24.107930 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.107905    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ad9f28-88ec-450e-95fa-0e812c2de534-openshift-service-ca\") pod \"b9ad9f28-88ec-450e-95fa-0e812c2de534\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") "
Sep 18 21:33:24.108044 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.107997    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbx82\" (UniqueName: \"kubernetes.io/projected/b9ad9f28-88ec-450e-95fa-0e812c2de534-kube-api-access-vbx82\") pod \"b9ad9f28-88ec-450e-95fa-0e812c2de534\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") "
Sep 18 21:33:24.108044 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.108031    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9ad9f28-88ec-450e-95fa-0e812c2de534-metrics-certs\") pod \"b9ad9f28-88ec-450e-95fa-0e812c2de534\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") "
Sep 18 21:33:24.108151 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.108060    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ad9f28-88ec-450e-95fa-0e812c2de534-maas-api-tls\") pod \"b9ad9f28-88ec-450e-95fa-0e812c2de534\" (UID: \"b9ad9f28-88ec-450e-95fa-0e812c2de534\") "
Sep 18 21:33:24.108608 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.108270    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ad9f28-88ec-450e-95fa-0e812c2de534-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9ad9f28-88ec-450e-95fa-0e812c2de534" (UID: "b9ad9f28-88ec-450e-95fa-0e812c2de534"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:33:24.112690 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.111003    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ad9f28-88ec-450e-95fa-0e812c2de534-kube-api-access-vbx82" (OuterVolumeSpecName: "kube-api-access-vbx82") pod "b9ad9f28-88ec-450e-95fa-0e812c2de534" (UID: "b9ad9f28-88ec-450e-95fa-0e812c2de534"). InnerVolumeSpecName "kube-api-access-vbx82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:33:24.112690 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.111027    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9ad9f28-88ec-450e-95fa-0e812c2de534-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b9ad9f28-88ec-450e-95fa-0e812c2de534" (UID: "b9ad9f28-88ec-450e-95fa-0e812c2de534"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:33:24.117034 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.117003    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9ad9f28-88ec-450e-95fa-0e812c2de534-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9ad9f28-88ec-450e-95fa-0e812c2de534" (UID: "b9ad9f28-88ec-450e-95fa-0e812c2de534"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:33:24.210134 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.210102    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbx82\" (UniqueName: \"kubernetes.io/projected/b9ad9f28-88ec-450e-95fa-0e812c2de534-kube-api-access-vbx82\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:24.210134 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.210128    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9ad9f28-88ec-450e-95fa-0e812c2de534-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:24.210134 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.210138    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9ad9f28-88ec-450e-95fa-0e812c2de534-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:24.210357 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.210146    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ad9f28-88ec-450e-95fa-0e812c2de534-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:24.218091 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.218055    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:24.374907 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.374833    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:24.581096 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.581063    2579 generic.go:356] "Generic (PLEG): container finished" podID="b9ad9f28-88ec-450e-95fa-0e812c2de534" containerID="abe8c51ef42f1096eded7dbe412bdc969815547f4f2f2cc97b67631110cef8c4" exitCode=0
Sep 18 21:33:24.581250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.581123    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"
Sep 18 21:33:24.581250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.581146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh" event={"ID":"b9ad9f28-88ec-450e-95fa-0e812c2de534","Type":"ContainerDied","Data":"abe8c51ef42f1096eded7dbe412bdc969815547f4f2f2cc97b67631110cef8c4"}
Sep 18 21:33:24.581250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.581219    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh" event={"ID":"b9ad9f28-88ec-450e-95fa-0e812c2de534","Type":"ContainerDied","Data":"b08a4eb77e608561d4ecb90416e0013092d7099310bc8995ad641b93d38e7e59"}
Sep 18 21:33:24.581250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.581241    2579 scope.go:117] "RemoveContainer" containerID="abe8c51ef42f1096eded7dbe412bdc969815547f4f2f2cc97b67631110cef8c4"
Sep 18 21:33:24.591349 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.591328    2579 scope.go:117] "RemoveContainer" containerID="abe8c51ef42f1096eded7dbe412bdc969815547f4f2f2cc97b67631110cef8c4"
Sep 18 21:33:24.591584 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:33:24.591567    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe8c51ef42f1096eded7dbe412bdc969815547f4f2f2cc97b67631110cef8c4\": container with ID starting with abe8c51ef42f1096eded7dbe412bdc969815547f4f2f2cc97b67631110cef8c4 not found: ID does not exist" containerID="abe8c51ef42f1096eded7dbe412bdc969815547f4f2f2cc97b67631110cef8c4"
Sep 18 21:33:24.591634 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.591592    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe8c51ef42f1096eded7dbe412bdc969815547f4f2f2cc97b67631110cef8c4"} err="failed to get container status \"abe8c51ef42f1096eded7dbe412bdc969815547f4f2f2cc97b67631110cef8c4\": rpc error: code = NotFound desc = could not find container \"abe8c51ef42f1096eded7dbe412bdc969815547f4f2f2cc97b67631110cef8c4\": container with ID starting with abe8c51ef42f1096eded7dbe412bdc969815547f4f2f2cc97b67631110cef8c4 not found: ID does not exist"
Sep 18 21:33:24.609050 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.609026    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"]
Sep 18 21:33:24.613533 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.613513    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh"]
Sep 18 21:33:24.640551 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.640501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:24.738217 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.738195    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:24.923529 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:24.923468    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9ad9f28-88ec-450e-95fa-0e812c2de534" path="/var/lib/kubelet/pods/b9ad9f28-88ec-450e-95fa-0e812c2de534/volumes"
Sep 18 21:33:25.753366 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:25.753333    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:26.629853 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:26.629817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:27.058123 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:27.058089    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1b65ac-64bbfcb45d-5lfrh" podUID="b9ad9f28-88ec-450e-95fa-0e812c2de534" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.58:8443/health\": context deadline exceeded"
Sep 18 21:33:27.517925 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:27.517810    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:30.406290 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:30.406257    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:30.504402 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:30.504370    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:30.683610 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:30.683528    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:31.293440 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:31.293402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:32.132287 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:32.132254    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:33.001460 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:33.001430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:34.001405 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:34.001368    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:34.774187 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:34.774153    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:35.687753 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:35.687722    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:37.877321 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:37.877282    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:38.869250 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:38.869208    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"]
Sep 18 21:33:38.892293 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:38.892252    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:41.129429 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:41.129393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:41.707265 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:41.707234    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"]
Sep 18 21:33:41.707514 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:41.707483    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc" podUID="97187b4f-0659-401e-956a-0da472757be4" containerName="maas-api" containerID="cri-o://e8cac72c7b437a4f24525f099166c1b146c661885e7806c71fa93b99f92de8c3" gracePeriod=30
Sep 18 21:33:41.744077 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:41.744040    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:41.813428 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:41.813356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:41.891288 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:41.891251    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:41.964791 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:41.964730    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:33:41.988426 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:41.988397    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:42.058707 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.058681    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97187b4f-0659-401e-956a-0da472757be4-openshift-service-ca\") pod \"97187b4f-0659-401e-956a-0da472757be4\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") "
Sep 18 21:33:42.058865 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.058723    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frsv9\" (UniqueName: \"kubernetes.io/projected/97187b4f-0659-401e-956a-0da472757be4-kube-api-access-frsv9\") pod \"97187b4f-0659-401e-956a-0da472757be4\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") "
Sep 18 21:33:42.058865 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.058766    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97187b4f-0659-401e-956a-0da472757be4-maas-api-tls\") pod \"97187b4f-0659-401e-956a-0da472757be4\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") "
Sep 18 21:33:42.058865 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.058861    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97187b4f-0659-401e-956a-0da472757be4-metrics-certs\") pod \"97187b4f-0659-401e-956a-0da472757be4\" (UID: \"97187b4f-0659-401e-956a-0da472757be4\") "
Sep 18 21:33:42.059095 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.059069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97187b4f-0659-401e-956a-0da472757be4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97187b4f-0659-401e-956a-0da472757be4" (UID: "97187b4f-0659-401e-956a-0da472757be4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:33:42.059274 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.059252    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97187b4f-0659-401e-956a-0da472757be4-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:42.060972 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.060939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97187b4f-0659-401e-956a-0da472757be4-kube-api-access-frsv9" (OuterVolumeSpecName: "kube-api-access-frsv9") pod "97187b4f-0659-401e-956a-0da472757be4" (UID: "97187b4f-0659-401e-956a-0da472757be4"). InnerVolumeSpecName "kube-api-access-frsv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:33:42.060972 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.060950    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97187b4f-0659-401e-956a-0da472757be4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97187b4f-0659-401e-956a-0da472757be4" (UID: "97187b4f-0659-401e-956a-0da472757be4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:33:42.061104 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.060991    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97187b4f-0659-401e-956a-0da472757be4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "97187b4f-0659-401e-956a-0da472757be4" (UID: "97187b4f-0659-401e-956a-0da472757be4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:33:42.160723 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.160683    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97187b4f-0659-401e-956a-0da472757be4-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:42.160723 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.160715    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97187b4f-0659-401e-956a-0da472757be4-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:42.160723 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.160725    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frsv9\" (UniqueName: \"kubernetes.io/projected/97187b4f-0659-401e-956a-0da472757be4-kube-api-access-frsv9\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:42.311746 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.311714    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:42.421278 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.421245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:42.680882 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.680797    2579 generic.go:356] "Generic (PLEG): container finished" podID="97187b4f-0659-401e-956a-0da472757be4" containerID="e8cac72c7b437a4f24525f099166c1b146c661885e7806c71fa93b99f92de8c3" exitCode=0
Sep 18 21:33:42.680882 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.680854    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc" event={"ID":"97187b4f-0659-401e-956a-0da472757be4","Type":"ContainerDied","Data":"e8cac72c7b437a4f24525f099166c1b146c661885e7806c71fa93b99f92de8c3"}
Sep 18 21:33:42.681093 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.680906    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"
Sep 18 21:33:42.681093 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.680922    2579 scope.go:117] "RemoveContainer" containerID="e8cac72c7b437a4f24525f099166c1b146c661885e7806c71fa93b99f92de8c3"
Sep 18 21:33:42.681093 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.680909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc" event={"ID":"97187b4f-0659-401e-956a-0da472757be4","Type":"ContainerDied","Data":"00fa67b41482702806f8c958f25f5be94e73c6bc3a191433049a64cbb5167759"}
Sep 18 21:33:42.691135 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.691066    2579 scope.go:117] "RemoveContainer" containerID="e8cac72c7b437a4f24525f099166c1b146c661885e7806c71fa93b99f92de8c3"
Sep 18 21:33:42.691401 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:33:42.691376    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cac72c7b437a4f24525f099166c1b146c661885e7806c71fa93b99f92de8c3\": container with ID starting with e8cac72c7b437a4f24525f099166c1b146c661885e7806c71fa93b99f92de8c3 not found: ID does not exist" containerID="e8cac72c7b437a4f24525f099166c1b146c661885e7806c71fa93b99f92de8c3"
Sep 18 21:33:42.691451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.691410    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cac72c7b437a4f24525f099166c1b146c661885e7806c71fa93b99f92de8c3"} err="failed to get container status \"e8cac72c7b437a4f24525f099166c1b146c661885e7806c71fa93b99f92de8c3\": rpc error: code = NotFound desc = could not find container \"e8cac72c7b437a4f24525f099166c1b146c661885e7806c71fa93b99f92de8c3\": container with ID starting with e8cac72c7b437a4f24525f099166c1b146c661885e7806c71fa93b99f92de8c3 not found: ID does not exist"
Sep 18 21:33:42.703695 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.703670    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"]
Sep 18 21:33:42.707668 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.707627    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-514a1a-584ddfdb5d-5ssrc"]
Sep 18 21:33:42.922088 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:42.922061    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97187b4f-0659-401e-956a-0da472757be4" path="/var/lib/kubelet/pods/97187b4f-0659-401e-956a-0da472757be4/volumes"
Sep 18 21:33:43.026155 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:43.026065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:43.221743 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:43.221712    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:43.921200 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:43.921167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:49.311354 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:49.311323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:49.412186 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:49.412155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:49.544434 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:49.544403    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"]
Sep 18 21:33:49.544739 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:49.544679    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy" containerID="cri-o://43a35632b87c08bd4a8fc20bbd9b5b9b00fcac848cbb09e92684ac59afdae90a" gracePeriod=30
Sep 18 21:33:49.575127 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:49.575060    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:33:53.890517 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:53.890481    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb" podUID="f10c3a3c-6955-482b-a20d-010c5c343ea3" containerName="main" containerID="cri-o://214b1b169e52ed82a9fdd8d26fc0392fafd1969c33fee25b74aa18bbea029c16" gracePeriod=45
Sep 18 21:33:54.133898 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.133876    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:33:54.269927 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.269864    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw4tv\" (UniqueName: \"kubernetes.io/projected/f10c3a3c-6955-482b-a20d-010c5c343ea3-kube-api-access-jw4tv\") pod \"f10c3a3c-6955-482b-a20d-010c5c343ea3\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") "
Sep 18 21:33:54.270052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.269948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-home\") pod \"f10c3a3c-6955-482b-a20d-010c5c343ea3\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") "
Sep 18 21:33:54.270052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.269971    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10c3a3c-6955-482b-a20d-010c5c343ea3-tls-certs\") pod \"f10c3a3c-6955-482b-a20d-010c5c343ea3\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") "
Sep 18 21:33:54.270052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.270007    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-tmp-dir\") pod \"f10c3a3c-6955-482b-a20d-010c5c343ea3\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") "
Sep 18 21:33:54.270052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.270021    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-dshm\") pod \"f10c3a3c-6955-482b-a20d-010c5c343ea3\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") "
Sep 18 21:33:54.270052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.270043    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-model-cache\") pod \"f10c3a3c-6955-482b-a20d-010c5c343ea3\" (UID: \"f10c3a3c-6955-482b-a20d-010c5c343ea3\") "
Sep 18 21:33:54.270325 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.270176    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-home" (OuterVolumeSpecName: "home") pod "f10c3a3c-6955-482b-a20d-010c5c343ea3" (UID: "f10c3a3c-6955-482b-a20d-010c5c343ea3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:33:54.270325 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.270304    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f10c3a3c-6955-482b-a20d-010c5c343ea3" (UID: "f10c3a3c-6955-482b-a20d-010c5c343ea3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:33:54.270422 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.270339    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f10c3a3c-6955-482b-a20d-010c5c343ea3" (UID: "f10c3a3c-6955-482b-a20d-010c5c343ea3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:33:54.270422 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.270348    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-home\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.272019 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.271996    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-dshm" (OuterVolumeSpecName: "dshm") pod "f10c3a3c-6955-482b-a20d-010c5c343ea3" (UID: "f10c3a3c-6955-482b-a20d-010c5c343ea3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:33:54.272019 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.272012    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10c3a3c-6955-482b-a20d-010c5c343ea3-kube-api-access-jw4tv" (OuterVolumeSpecName: "kube-api-access-jw4tv") pod "f10c3a3c-6955-482b-a20d-010c5c343ea3" (UID: "f10c3a3c-6955-482b-a20d-010c5c343ea3"). InnerVolumeSpecName "kube-api-access-jw4tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:33:54.272149 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.272060    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10c3a3c-6955-482b-a20d-010c5c343ea3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f10c3a3c-6955-482b-a20d-010c5c343ea3" (UID: "f10c3a3c-6955-482b-a20d-010c5c343ea3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:33:54.370996 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.370972    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw4tv\" (UniqueName: \"kubernetes.io/projected/f10c3a3c-6955-482b-a20d-010c5c343ea3-kube-api-access-jw4tv\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.370996 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.370995    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f10c3a3c-6955-482b-a20d-010c5c343ea3-tls-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.371136 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.371006    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-tmp-dir\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.371136 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.371014    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-dshm\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.371136 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.371022    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f10c3a3c-6955-482b-a20d-010c5c343ea3-model-cache\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.746939 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.746910    2579 generic.go:356] "Generic (PLEG): container finished" podID="f10c3a3c-6955-482b-a20d-010c5c343ea3" containerID="214b1b169e52ed82a9fdd8d26fc0392fafd1969c33fee25b74aa18bbea029c16" exitCode=0
Sep 18 21:33:54.747131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.746958    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"
Sep 18 21:33:54.747131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.746994    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb" event={"ID":"f10c3a3c-6955-482b-a20d-010c5c343ea3","Type":"ContainerDied","Data":"214b1b169e52ed82a9fdd8d26fc0392fafd1969c33fee25b74aa18bbea029c16"}
Sep 18 21:33:54.747131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.747040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb" event={"ID":"f10c3a3c-6955-482b-a20d-010c5c343ea3","Type":"ContainerDied","Data":"ee15b27f2f0cd9b70aee15eb072e251d4509d46c0b5ad66d425949ca10edfb3a"}
Sep 18 21:33:54.747131 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.747063    2579 scope.go:117] "RemoveContainer" containerID="214b1b169e52ed82a9fdd8d26fc0392fafd1969c33fee25b74aa18bbea029c16"
Sep 18 21:33:54.749404 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.749381    2579 generic.go:356] "Generic (PLEG): container finished" podID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerID="43a35632b87c08bd4a8fc20bbd9b5b9b00fcac848cbb09e92684ac59afdae90a" exitCode=0
Sep 18 21:33:54.749547 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.749421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" event={"ID":"278bea04-a288-4f2b-8686-c444ec8d4b43","Type":"ContainerDied","Data":"43a35632b87c08bd4a8fc20bbd9b5b9b00fcac848cbb09e92684ac59afdae90a"}
Sep 18 21:33:54.756161 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.756131    2579 scope.go:117] "RemoveContainer" containerID="214b1b169e52ed82a9fdd8d26fc0392fafd1969c33fee25b74aa18bbea029c16"
Sep 18 21:33:54.756439 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:33:54.756414    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214b1b169e52ed82a9fdd8d26fc0392fafd1969c33fee25b74aa18bbea029c16\": container with ID starting with 214b1b169e52ed82a9fdd8d26fc0392fafd1969c33fee25b74aa18bbea029c16 not found: ID does not exist" containerID="214b1b169e52ed82a9fdd8d26fc0392fafd1969c33fee25b74aa18bbea029c16"
Sep 18 21:33:54.756510 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.756446    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214b1b169e52ed82a9fdd8d26fc0392fafd1969c33fee25b74aa18bbea029c16"} err="failed to get container status \"214b1b169e52ed82a9fdd8d26fc0392fafd1969c33fee25b74aa18bbea029c16\": rpc error: code = NotFound desc = could not find container \"214b1b169e52ed82a9fdd8d26fc0392fafd1969c33fee25b74aa18bbea029c16\": container with ID starting with 214b1b169e52ed82a9fdd8d26fc0392fafd1969c33fee25b74aa18bbea029c16 not found: ID does not exist"
Sep 18 21:33:54.756510 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.756463    2579 scope.go:117] "RemoveContainer" containerID="5d5eb188ecd608a8b31ea7e61a9b56920f867d4862bf2b2167e460b6c79049ea"
Sep 18 21:33:54.771171 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.771147    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"]
Sep 18 21:33:54.784200 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.784181    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/sub-test-model-fd993e-kserve-7f87f8b855-dn7rb"]
Sep 18 21:33:54.785638 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.785623    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:33:54.875752 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.875680    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-podinfo\") pod \"278bea04-a288-4f2b-8686-c444ec8d4b43\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") "
Sep 18 21:33:54.875860 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.875775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59mqh\" (UniqueName: \"kubernetes.io/projected/278bea04-a288-4f2b-8686-c444ec8d4b43-kube-api-access-59mqh\") pod \"278bea04-a288-4f2b-8686-c444ec8d4b43\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") "
Sep 18 21:33:54.875860 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.875806    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-workload-socket\") pod \"278bea04-a288-4f2b-8686-c444ec8d4b43\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") "
Sep 18 21:33:54.875860 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.875851    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-data\") pod \"278bea04-a288-4f2b-8686-c444ec8d4b43\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") "
Sep 18 21:33:54.875989 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.875896    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278bea04-a288-4f2b-8686-c444ec8d4b43-istiod-ca-cert\") pod \"278bea04-a288-4f2b-8686-c444ec8d4b43\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") "
Sep 18 21:33:54.875989 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.875928    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-envoy\") pod \"278bea04-a288-4f2b-8686-c444ec8d4b43\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") "
Sep 18 21:33:54.875989 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.875953    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-workload-certs\") pod \"278bea04-a288-4f2b-8686-c444ec8d4b43\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") "
Sep 18 21:33:54.875989 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.875975    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-token\") pod \"278bea04-a288-4f2b-8686-c444ec8d4b43\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") "
Sep 18 21:33:54.876208 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.876002    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-credential-socket\") pod \"278bea04-a288-4f2b-8686-c444ec8d4b43\" (UID: \"278bea04-a288-4f2b-8686-c444ec8d4b43\") "
Sep 18 21:33:54.876208 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.876113    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "278bea04-a288-4f2b-8686-c444ec8d4b43" (UID: "278bea04-a288-4f2b-8686-c444ec8d4b43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:33:54.876322 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.876201    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-data" (OuterVolumeSpecName: "istio-data") pod "278bea04-a288-4f2b-8686-c444ec8d4b43" (UID: "278bea04-a288-4f2b-8686-c444ec8d4b43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:33:54.876322 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.876291    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278bea04-a288-4f2b-8686-c444ec8d4b43-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "278bea04-a288-4f2b-8686-c444ec8d4b43" (UID: "278bea04-a288-4f2b-8686-c444ec8d4b43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:33:54.876442 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.876368    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-workload-socket\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.876442 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.876388    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-data\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.876442 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.876411    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "278bea04-a288-4f2b-8686-c444ec8d4b43" (UID: "278bea04-a288-4f2b-8686-c444ec8d4b43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:33:54.876442 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.876426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "278bea04-a288-4f2b-8686-c444ec8d4b43" (UID: "278bea04-a288-4f2b-8686-c444ec8d4b43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:33:54.878104 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.878083    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-token" (OuterVolumeSpecName: "istio-token") pod "278bea04-a288-4f2b-8686-c444ec8d4b43" (UID: "278bea04-a288-4f2b-8686-c444ec8d4b43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:33:54.878181 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.878116    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "278bea04-a288-4f2b-8686-c444ec8d4b43" (UID: "278bea04-a288-4f2b-8686-c444ec8d4b43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:33:54.878340 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.878313    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "278bea04-a288-4f2b-8686-c444ec8d4b43" (UID: "278bea04-a288-4f2b-8686-c444ec8d4b43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 21:33:54.878401 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.878347    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278bea04-a288-4f2b-8686-c444ec8d4b43-kube-api-access-59mqh" (OuterVolumeSpecName: "kube-api-access-59mqh") pod "278bea04-a288-4f2b-8686-c444ec8d4b43" (UID: "278bea04-a288-4f2b-8686-c444ec8d4b43"). InnerVolumeSpecName "kube-api-access-59mqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:33:54.923033 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.923005    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f10c3a3c-6955-482b-a20d-010c5c343ea3" path="/var/lib/kubelet/pods/f10c3a3c-6955-482b-a20d-010c5c343ea3/volumes"
Sep 18 21:33:54.977611 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.977571    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278bea04-a288-4f2b-8686-c444ec8d4b43-istiod-ca-cert\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.977611 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.977605    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-envoy\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.977611 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.977617    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-workload-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.977853 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.977627    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-token\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.977853 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.977638    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bea04-a288-4f2b-8686-c444ec8d4b43-credential-socket\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.977853 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.977664    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278bea04-a288-4f2b-8686-c444ec8d4b43-istio-podinfo\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:54.977853 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:54.977676    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59mqh\" (UniqueName: \"kubernetes.io/projected/278bea04-a288-4f2b-8686-c444ec8d4b43-kube-api-access-59mqh\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:33:55.756740 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:55.756714    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"
Sep 18 21:33:55.756922 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:55.756762    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n" event={"ID":"278bea04-a288-4f2b-8686-c444ec8d4b43","Type":"ContainerDied","Data":"2215ca798a038f5042382b79b5a1036d55bb1cfffdf2e50807601daf99526ed2"}
Sep 18 21:33:55.756922 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:55.756833    2579 scope.go:117] "RemoveContainer" containerID="43a35632b87c08bd4a8fc20bbd9b5b9b00fcac848cbb09e92684ac59afdae90a"
Sep 18 21:33:55.788158 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:55.788136    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"]
Sep 18 21:33:55.798602 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:55.798582    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-514a1a-openshift-default-76bd5c667d-xz26n"]
Sep 18 21:33:56.921608 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:33:56.921576    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" path="/var/lib/kubelet/pods/278bea04-a288-4f2b-8686-c444ec8d4b43/volumes"
Sep 18 21:34:01.630133 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:01.630088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:34:07.619244 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:07.619206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:34:42.673002 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.672959    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"]
Sep 18 21:34:42.673571 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673550    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy"
Sep 18 21:34:42.673571 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673570    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy"
Sep 18 21:34:42.673571 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673581    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ad9f28-88ec-450e-95fa-0e812c2de534" containerName="maas-api"
Sep 18 21:34:42.673773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673589    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ad9f28-88ec-450e-95fa-0e812c2de534" containerName="maas-api"
Sep 18 21:34:42.673773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673599    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5903bc-60a2-45bb-a8a5-1c48b4b5613d" containerName="maas-api"
Sep 18 21:34:42.673773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673605    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5903bc-60a2-45bb-a8a5-1c48b4b5613d" containerName="maas-api"
Sep 18 21:34:42.673773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673623    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy"
Sep 18 21:34:42.673773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673628    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy"
Sep 18 21:34:42.673773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673635    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97187b4f-0659-401e-956a-0da472757be4" containerName="maas-api"
Sep 18 21:34:42.673773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673653    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="97187b4f-0659-401e-956a-0da472757be4" containerName="maas-api"
Sep 18 21:34:42.673773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673680    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f10c3a3c-6955-482b-a20d-010c5c343ea3" containerName="main"
Sep 18 21:34:42.673773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673686    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f10c3a3c-6955-482b-a20d-010c5c343ea3" containerName="main"
Sep 18 21:34:42.673773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673758    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="97187b4f-0659-401e-956a-0da472757be4" containerName="maas-api"
Sep 18 21:34:42.673773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673765    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy"
Sep 18 21:34:42.673773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673771    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef5903bc-60a2-45bb-a8a5-1c48b4b5613d" containerName="maas-api"
Sep 18 21:34:42.673773 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673778    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f10c3a3c-6955-482b-a20d-010c5c343ea3" containerName="main"
Sep 18 21:34:42.674158 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.673787    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ad9f28-88ec-450e-95fa-0e812c2de534" containerName="maas-api"
Sep 18 21:34:42.677032 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.677011    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.680782 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.680758    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd993e\"/\"rate-test-model-fd993e-kserve-self-signed-certs\""
Sep 18 21:34:42.680901 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.680763    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd993e\"/\"openshift-service-ca.crt\""
Sep 18 21:34:42.680901 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.680769    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd993e\"/\"default-dockercfg-p7k9z\""
Sep 18 21:34:42.680901 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.680816    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-fd993e\"/\"kube-root-ca.crt\""
Sep 18 21:34:42.686320 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.686298    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"]
Sep 18 21:34:42.705149 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.705110    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:34:42.818896 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.818848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-home\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.818896 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.818911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-tmp-dir\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.819140 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.818957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-dshm\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.819140 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.818975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-model-cache\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.819140 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.819043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j8dq\" (UniqueName: \"kubernetes.io/projected/5b06ee4d-e757-4d12-9c97-62521259c81c-kube-api-access-8j8dq\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.819140 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.819125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b06ee4d-e757-4d12-9c97-62521259c81c-tls-certs\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.919615 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.919579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-home\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.919872 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.919634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-tmp-dir\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.919872 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.919687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-dshm\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.919872 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.919714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-model-cache\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.919872 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.919765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j8dq\" (UniqueName: \"kubernetes.io/projected/5b06ee4d-e757-4d12-9c97-62521259c81c-kube-api-access-8j8dq\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.919872 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.919835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b06ee4d-e757-4d12-9c97-62521259c81c-tls-certs\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.920165 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.920076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-home\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.920214 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.920168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-model-cache\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.920285 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.920268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-tmp-dir\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.922381 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.922358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b06ee4d-e757-4d12-9c97-62521259c81c-tls-certs\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.922502 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.922360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-dshm\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.929567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.929521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j8dq\" (UniqueName: \"kubernetes.io/projected/5b06ee4d-e757-4d12-9c97-62521259c81c-kube-api-access-8j8dq\") pod \"rate-test-model-fd993e-kserve-8584bdf799-m5vd6\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") " pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:42.987029 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:42.987001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:43.262238 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:43.262151    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"]
Sep 18 21:34:43.264602 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:34:43.264570    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b06ee4d_e757_4d12_9c97_62521259c81c.slice/crio-f185fc8b451973b2c382f16baa4b04fa04bb5392b7759f042d80e440b2d421a4 WatchSource:0}: Error finding container f185fc8b451973b2c382f16baa4b04fa04bb5392b7759f042d80e440b2d421a4: Status 404 returned error can't find the container with id f185fc8b451973b2c382f16baa4b04fa04bb5392b7759f042d80e440b2d421a4
Sep 18 21:34:43.994427 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:43.994391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6" event={"ID":"5b06ee4d-e757-4d12-9c97-62521259c81c","Type":"ContainerStarted","Data":"6aa29d08e78bcedf5f26d5d40153ff991888c8b6cdb944c6644fdd214e4bafbd"}
Sep 18 21:34:43.994809 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:43.994433    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6" event={"ID":"5b06ee4d-e757-4d12-9c97-62521259c81c","Type":"ContainerStarted","Data":"f185fc8b451973b2c382f16baa4b04fa04bb5392b7759f042d80e440b2d421a4"}
Sep 18 21:34:44.016623 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:44.016573    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6" podStartSLOduration=2.016557697 podStartE2EDuration="2.016557697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:34:44.014508626 +0000 UTC m=+1721.621568185" watchObservedRunningTime="2026-09-18 21:34:44.016557697 +0000 UTC m=+1721.623617254"
Sep 18 21:34:52.987164 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:52.987115    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:52.987830 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:52.987192    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:53.002564 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:53.002533    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:34:53.051760 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:34:53.051729    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:35:33.686105 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:33.686069    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:34.571409 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:34.571369    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"]
Sep 18 21:35:34.593797 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:34.593770    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:35.054363 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:35.054332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:35.381137 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:35.381063    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:35.750507 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:35.750430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:38.500597 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:38.500565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:38.595425 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:38.595398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:40.122336 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:40.122254    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:41.101784 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.101749    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"]
Sep 18 21:35:41.102214 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.102201    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="278bea04-a288-4f2b-8686-c444ec8d4b43" containerName="istio-proxy"
Sep 18 21:35:41.104820 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.104797    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:41.107520 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.107499    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5888e6-serving-cert\""
Sep 18 21:35:41.116418 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.116395    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"]
Sep 18 21:35:41.132852 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.132819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:41.155701 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.155678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/908b4416-b80f-4ed4-9591-e550ebc15658-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:41.155802 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.155709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:41.155802 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.155747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dw9r\" (UniqueName: \"kubernetes.io/projected/908b4416-b80f-4ed4-9591-e550ebc15658-kube-api-access-4dw9r\") pod \"maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:41.155895 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.155877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-metrics-certs\") pod \"maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:41.232594 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.232572    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:41.256941 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.256917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-metrics-certs\") pod \"maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:41.257028 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.256968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/908b4416-b80f-4ed4-9591-e550ebc15658-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:41.257028 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.256986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:41.257028 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.257017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dw9r\" (UniqueName: \"kubernetes.io/projected/908b4416-b80f-4ed4-9591-e550ebc15658-kube-api-access-4dw9r\") pod \"maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:41.257153 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:35:41.257137    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-serving-cert: secret "maas-api-e2e-ipp-a-5888e6-serving-cert" not found
Sep 18 21:35:41.257221 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:35:41.257207    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-maas-api-tls podName:908b4416-b80f-4ed4-9591-e550ebc15658 nodeName:}" failed. No retries permitted until 2026-09-18 21:35:41.757187973 +0000 UTC m=+1779.364247513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-maas-api-tls") pod "maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9" (UID: "908b4416-b80f-4ed4-9591-e550ebc15658") : secret "maas-api-e2e-ipp-a-5888e6-serving-cert" not found
Sep 18 21:35:41.257526 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.257507    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/908b4416-b80f-4ed4-9591-e550ebc15658-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:41.259727 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.259704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-metrics-certs\") pod \"maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:41.267315 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.267292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dw9r\" (UniqueName: \"kubernetes.io/projected/908b4416-b80f-4ed4-9591-e550ebc15658-kube-api-access-4dw9r\") pod \"maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:41.333374 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.333347    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:41.611539 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.611507    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:41.701887 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.701850    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:41.761543 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.761514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:41.763835 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:41.763807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:42.016530 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:42.016449    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:42.290678 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:42.290636    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"]
Sep 18 21:35:42.293121 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:35:42.293091    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908b4416_b80f_4ed4_9591_e550ebc15658.slice/crio-097b0bf4fe00702e478b4d5182eb4c0f44daa5cd5b62ecf18465fed580b789a1 WatchSource:0}: Error finding container 097b0bf4fe00702e478b4d5182eb4c0f44daa5cd5b62ecf18465fed580b789a1: Status 404 returned error can't find the container with id 097b0bf4fe00702e478b4d5182eb4c0f44daa5cd5b62ecf18465fed580b789a1
Sep 18 21:35:42.295240 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:42.295215    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:35:43.290980 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:43.290941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9" event={"ID":"908b4416-b80f-4ed4-9591-e550ebc15658","Type":"ContainerStarted","Data":"74a39bf137c4a190fa1230e11f55c4f5bc597a0116441934612697b26e02257b"}
Sep 18 21:35:43.291427 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:43.290996    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9" event={"ID":"908b4416-b80f-4ed4-9591-e550ebc15658","Type":"ContainerStarted","Data":"097b0bf4fe00702e478b4d5182eb4c0f44daa5cd5b62ecf18465fed580b789a1"}
Sep 18 21:35:43.291427 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:43.291019    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:43.310371 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:43.310328    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9" podStartSLOduration=1.88000496 podStartE2EDuration="2.310313554s" totalImagesPullingTime="430.308594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:35:43.308381053 +0000 UTC m=+1780.915440626" watchObservedRunningTime="2026-09-18 21:35:43.310313554 +0000 UTC m=+1780.917373133"
Sep 18 21:35:49.302249 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.302219    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:35:49.590639 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.590539    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6" podUID="5b06ee4d-e757-4d12-9c97-62521259c81c" containerName="main" containerID="cri-o://6aa29d08e78bcedf5f26d5d40153ff991888c8b6cdb944c6644fdd214e4bafbd" gracePeriod=45
Sep 18 21:35:49.835612 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.835589    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:35:49.935832 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.935764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-dshm\") pod \"5b06ee4d-e757-4d12-9c97-62521259c81c\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") "
Sep 18 21:35:49.935832 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.935801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-model-cache\") pod \"5b06ee4d-e757-4d12-9c97-62521259c81c\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") "
Sep 18 21:35:49.936041 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.935840    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b06ee4d-e757-4d12-9c97-62521259c81c-tls-certs\") pod \"5b06ee4d-e757-4d12-9c97-62521259c81c\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") "
Sep 18 21:35:49.936041 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.935867    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-home\") pod \"5b06ee4d-e757-4d12-9c97-62521259c81c\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") "
Sep 18 21:35:49.936041 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.935891    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j8dq\" (UniqueName: \"kubernetes.io/projected/5b06ee4d-e757-4d12-9c97-62521259c81c-kube-api-access-8j8dq\") pod \"5b06ee4d-e757-4d12-9c97-62521259c81c\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") "
Sep 18 21:35:49.936041 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.935932    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-tmp-dir\") pod \"5b06ee4d-e757-4d12-9c97-62521259c81c\" (UID: \"5b06ee4d-e757-4d12-9c97-62521259c81c\") "
Sep 18 21:35:49.936236 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.936062    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b06ee4d-e757-4d12-9c97-62521259c81c" (UID: "5b06ee4d-e757-4d12-9c97-62521259c81c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:35:49.936236 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.936106    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-home" (OuterVolumeSpecName: "home") pod "5b06ee4d-e757-4d12-9c97-62521259c81c" (UID: "5b06ee4d-e757-4d12-9c97-62521259c81c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:35:49.936339 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.936264    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b06ee4d-e757-4d12-9c97-62521259c81c" (UID: "5b06ee4d-e757-4d12-9c97-62521259c81c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:35:49.936339 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.936286    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-home\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:35:49.936339 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.936306    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-model-cache\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:35:49.937856 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.937833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b06ee4d-e757-4d12-9c97-62521259c81c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b06ee4d-e757-4d12-9c97-62521259c81c" (UID: "5b06ee4d-e757-4d12-9c97-62521259c81c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:35:49.937958 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.937875    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b06ee4d-e757-4d12-9c97-62521259c81c-kube-api-access-8j8dq" (OuterVolumeSpecName: "kube-api-access-8j8dq") pod "5b06ee4d-e757-4d12-9c97-62521259c81c" (UID: "5b06ee4d-e757-4d12-9c97-62521259c81c"). InnerVolumeSpecName "kube-api-access-8j8dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:35:49.937958 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:49.937902    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-dshm" (OuterVolumeSpecName: "dshm") pod "5b06ee4d-e757-4d12-9c97-62521259c81c" (UID: "5b06ee4d-e757-4d12-9c97-62521259c81c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:35:50.036772 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.036746    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-dshm\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:35:50.036772 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.036772    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b06ee4d-e757-4d12-9c97-62521259c81c-tls-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:35:50.036915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.036787    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j8dq\" (UniqueName: \"kubernetes.io/projected/5b06ee4d-e757-4d12-9c97-62521259c81c-kube-api-access-8j8dq\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:35:50.036915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.036800    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b06ee4d-e757-4d12-9c97-62521259c81c-tmp-dir\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:35:50.327525 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.327495    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b06ee4d-e757-4d12-9c97-62521259c81c" containerID="6aa29d08e78bcedf5f26d5d40153ff991888c8b6cdb944c6644fdd214e4bafbd" exitCode=0
Sep 18 21:35:50.327936 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.327534    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"
Sep 18 21:35:50.327936 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.327577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6" event={"ID":"5b06ee4d-e757-4d12-9c97-62521259c81c","Type":"ContainerDied","Data":"6aa29d08e78bcedf5f26d5d40153ff991888c8b6cdb944c6644fdd214e4bafbd"}
Sep 18 21:35:50.327936 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.327626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6" event={"ID":"5b06ee4d-e757-4d12-9c97-62521259c81c","Type":"ContainerDied","Data":"f185fc8b451973b2c382f16baa4b04fa04bb5392b7759f042d80e440b2d421a4"}
Sep 18 21:35:50.327936 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.327680    2579 scope.go:117] "RemoveContainer" containerID="6aa29d08e78bcedf5f26d5d40153ff991888c8b6cdb944c6644fdd214e4bafbd"
Sep 18 21:35:50.338139 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.338121    2579 scope.go:117] "RemoveContainer" containerID="6aa29d08e78bcedf5f26d5d40153ff991888c8b6cdb944c6644fdd214e4bafbd"
Sep 18 21:35:50.338448 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:35:50.338428    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa29d08e78bcedf5f26d5d40153ff991888c8b6cdb944c6644fdd214e4bafbd\": container with ID starting with 6aa29d08e78bcedf5f26d5d40153ff991888c8b6cdb944c6644fdd214e4bafbd not found: ID does not exist" containerID="6aa29d08e78bcedf5f26d5d40153ff991888c8b6cdb944c6644fdd214e4bafbd"
Sep 18 21:35:50.338509 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.338463    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa29d08e78bcedf5f26d5d40153ff991888c8b6cdb944c6644fdd214e4bafbd"} err="failed to get container status \"6aa29d08e78bcedf5f26d5d40153ff991888c8b6cdb944c6644fdd214e4bafbd\": rpc error: code = NotFound desc = could not find container \"6aa29d08e78bcedf5f26d5d40153ff991888c8b6cdb944c6644fdd214e4bafbd\": container with ID starting with 6aa29d08e78bcedf5f26d5d40153ff991888c8b6cdb944c6644fdd214e4bafbd not found: ID does not exist"
Sep 18 21:35:50.350454 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.350428    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"]
Sep 18 21:35:50.354521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.354500    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-fd993e/rate-test-model-fd993e-kserve-8584bdf799-m5vd6"]
Sep 18 21:35:50.921766 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:50.921734    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b06ee4d-e757-4d12-9c97-62521259c81c" path="/var/lib/kubelet/pods/5b06ee4d-e757-4d12-9c97-62521259c81c/volumes"
Sep 18 21:35:55.337079 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:55.336989    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:58.674281 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:58.674244    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:35:58.775431 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:35:58.775402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:02.963382 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:02.963354    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:36:02.967154 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:02.967132    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:36:02.969945 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:02.969928    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:36:02.973625 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:02.973608    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:36:03.479156 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:03.479122    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:03.871969 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:03.871939    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:04.129534 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:04.129451    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:04.460434 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:04.460348    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:04.555753 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:04.555714    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:04.655388 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:04.655355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:04.755691 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:04.755606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:05.363668 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:05.363616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:06.260955 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:06.260915    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:07.459343 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:07.459310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:07.987257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:07.987229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:08.305304 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:08.305267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:09.546268 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:09.546234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:10.582135 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:10.582106    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:11.408928 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:11.408893    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:12.318103 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.318072    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"]
Sep 18 21:36:12.318472 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.318458    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b06ee4d-e757-4d12-9c97-62521259c81c" containerName="main"
Sep 18 21:36:12.318472 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.318473    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b06ee4d-e757-4d12-9c97-62521259c81c" containerName="main"
Sep 18 21:36:12.318561 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.318553    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b06ee4d-e757-4d12-9c97-62521259c81c" containerName="main"
Sep 18 21:36:12.321547 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.321532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.324185 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.324161    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-deac7a-serving-cert\""
Sep 18 21:36:12.342727 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.342699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe5c426-17e6-4e28-9986-c2bc89b18587-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.342836 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.342766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpvzl\" (UniqueName: \"kubernetes.io/projected/ffe5c426-17e6-4e28-9986-c2bc89b18587-kube-api-access-kpvzl\") pod \"maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.342836 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.342792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffe5c426-17e6-4e28-9986-c2bc89b18587-metrics-certs\") pod \"maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.342836 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.342832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffe5c426-17e6-4e28-9986-c2bc89b18587-maas-api-tls\") pod \"maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.346409 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.346386    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"]
Sep 18 21:36:12.357257 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.357232    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:12.444302 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.444269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe5c426-17e6-4e28-9986-c2bc89b18587-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.444458 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.444336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpvzl\" (UniqueName: \"kubernetes.io/projected/ffe5c426-17e6-4e28-9986-c2bc89b18587-kube-api-access-kpvzl\") pod \"maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.444458 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.444363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffe5c426-17e6-4e28-9986-c2bc89b18587-metrics-certs\") pod \"maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.444458 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.444400    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffe5c426-17e6-4e28-9986-c2bc89b18587-maas-api-tls\") pod \"maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.445014 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.444984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe5c426-17e6-4e28-9986-c2bc89b18587-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.446803 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.446783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffe5c426-17e6-4e28-9986-c2bc89b18587-metrics-certs\") pod \"maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.446884 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.446817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffe5c426-17e6-4e28-9986-c2bc89b18587-maas-api-tls\") pod \"maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.459591 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.459564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpvzl\" (UniqueName: \"kubernetes.io/projected/ffe5c426-17e6-4e28-9986-c2bc89b18587-kube-api-access-kpvzl\") pod \"maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.468177 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.468152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:12.563060 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.563034    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:12.632280 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.632207    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:12.872716 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.872619    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:12.909793 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.909769    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"]
Sep 18 21:36:12.910757 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:36:12.910733    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe5c426_17e6_4e28_9986_c2bc89b18587.slice/crio-00a1c32bb5ccd36555c3b1307ef4dc16fc132d118d99c7b6922994dc0e66532e WatchSource:0}: Error finding container 00a1c32bb5ccd36555c3b1307ef4dc16fc132d118d99c7b6922994dc0e66532e: Status 404 returned error can't find the container with id 00a1c32bb5ccd36555c3b1307ef4dc16fc132d118d99c7b6922994dc0e66532e
Sep 18 21:36:12.962183 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:12.962157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:13.445573 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:13.445535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp" event={"ID":"ffe5c426-17e6-4e28-9986-c2bc89b18587","Type":"ContainerStarted","Data":"8469f1e0429c5ba5ef40cb5d7e6af81bbb8160fe663e7b75d3a2e29afdaddba9"}
Sep 18 21:36:13.446001 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:13.445598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp" event={"ID":"ffe5c426-17e6-4e28-9986-c2bc89b18587","Type":"ContainerStarted","Data":"00a1c32bb5ccd36555c3b1307ef4dc16fc132d118d99c7b6922994dc0e66532e"}
Sep 18 21:36:13.446001 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:13.445696    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:13.467843 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:13.467760    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp" podStartSLOduration=1.16400289 podStartE2EDuration="1.46774568s" totalImagesPullingTime="303.74279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:36:13.465749515 +0000 UTC m=+1811.072809066" watchObservedRunningTime="2026-09-18 21:36:13.46774568 +0000 UTC m=+1811.074805250"
Sep 18 21:36:19.463537 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:19.463505    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:36:36.224223 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:36.224186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:37.042288 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:37.042255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:37.856847 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:37.856808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:37.958113 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:37.958084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:38.944355 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:38.944319    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:39.666135 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:39.666099    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:40.521841 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:40.521801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:36:42.886111 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:42.886076    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9kv57_0392321b-d271-4c1c-beea-5c2e7d5802db/payload-processing/0.log"
Sep 18 21:36:50.620818 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:36:50.620777    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:19.107590 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:19.107553    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9kv57_0392321b-d271-4c1c-beea-5c2e7d5802db/payload-processing/0.log"
Sep 18 21:37:19.626412 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:19.626380    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:20.769286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:20.769253    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:22.935721 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:22.935687    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:23.186891 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:23.186671    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:23.283582 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:23.283533    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:23.972086 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:23.972050    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:24.719409 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:24.719356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:25.523696 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:25.523597    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:26.522453 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.522414    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"]
Sep 18 21:37:26.526788 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.526762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:26.530521 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.530498    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-b3d150-serving-cert\""
Sep 18 21:37:26.547271 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.547244    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"]
Sep 18 21:37:26.565699 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.565672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34451a00-43bd-42a6-af1d-0b9449492697-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:26.565831 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.565716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34451a00-43bd-42a6-af1d-0b9449492697-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:26.565831 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.565804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34451a00-43bd-42a6-af1d-0b9449492697-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:26.565914 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.565847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chh2\" (UniqueName: \"kubernetes.io/projected/34451a00-43bd-42a6-af1d-0b9449492697-kube-api-access-2chh2\") pod \"maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:26.587724 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.587686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:26.666769 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.666736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34451a00-43bd-42a6-af1d-0b9449492697-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:26.666939 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.666787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34451a00-43bd-42a6-af1d-0b9449492697-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:26.666939 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.666850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34451a00-43bd-42a6-af1d-0b9449492697-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:26.666939 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.666893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chh2\" (UniqueName: \"kubernetes.io/projected/34451a00-43bd-42a6-af1d-0b9449492697-kube-api-access-2chh2\") pod \"maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:26.667556 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.667534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34451a00-43bd-42a6-af1d-0b9449492697-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:26.669293 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.669271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34451a00-43bd-42a6-af1d-0b9449492697-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:26.669395 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.669321    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34451a00-43bd-42a6-af1d-0b9449492697-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:26.675121 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.675096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:26.677094 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.677072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chh2\" (UniqueName: \"kubernetes.io/projected/34451a00-43bd-42a6-af1d-0b9449492697-kube-api-access-2chh2\") pod \"maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:26.837692 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:26.837639    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:27.119228 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.119202    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"]
Sep 18 21:37:27.122053 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:37:27.122023    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34451a00_43bd_42a6_af1d_0b9449492697.slice/crio-5bf64edffd0244993c8bde0fec7d77c528a69da227392a2650e66154c596bc3f WatchSource:0}: Error finding container 5bf64edffd0244993c8bde0fec7d77c528a69da227392a2650e66154c596bc3f: Status 404 returned error can't find the container with id 5bf64edffd0244993c8bde0fec7d77c528a69da227392a2650e66154c596bc3f
Sep 18 21:37:27.268902 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.268854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:27.364040 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.364005    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:27.483066 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.482984    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"]
Sep 18 21:37:27.487953 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.487935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"
Sep 18 21:37:27.491567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.491546    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-b3d150-dockercfg-cl7wr\""
Sep 18 21:37:27.492013 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.491993    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-b3d150\""
Sep 18 21:37:27.509966 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.509941    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"]
Sep 18 21:37:27.576916 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.576889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99281f53-e477-4f7d-a616-95589bb5da49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64\" (UID: \"99281f53-e477-4f7d-a616-95589bb5da49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"
Sep 18 21:37:27.577265 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.577048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxw8x\" (UniqueName: \"kubernetes.io/projected/99281f53-e477-4f7d-a616-95589bb5da49-kube-api-access-pxw8x\") pod \"payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64\" (UID: \"99281f53-e477-4f7d-a616-95589bb5da49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"
Sep 18 21:37:27.678343 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.678310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxw8x\" (UniqueName: \"kubernetes.io/projected/99281f53-e477-4f7d-a616-95589bb5da49-kube-api-access-pxw8x\") pod \"payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64\" (UID: \"99281f53-e477-4f7d-a616-95589bb5da49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"
Sep 18 21:37:27.678513 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.678418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99281f53-e477-4f7d-a616-95589bb5da49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64\" (UID: \"99281f53-e477-4f7d-a616-95589bb5da49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"
Sep 18 21:37:27.679144 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.679122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99281f53-e477-4f7d-a616-95589bb5da49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64\" (UID: \"99281f53-e477-4f7d-a616-95589bb5da49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"
Sep 18 21:37:27.688577 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.688552    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxw8x\" (UniqueName: \"kubernetes.io/projected/99281f53-e477-4f7d-a616-95589bb5da49-kube-api-access-pxw8x\") pod \"payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64\" (UID: \"99281f53-e477-4f7d-a616-95589bb5da49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"
Sep 18 21:37:27.799275 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.799241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"
Sep 18 21:37:27.832903 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.832868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7" event={"ID":"34451a00-43bd-42a6-af1d-0b9449492697","Type":"ContainerStarted","Data":"faa2fad3dc322b2bb86f5ff0ff9ba88f8bef08c41b8564436e617e04d8eb64e9"}
Sep 18 21:37:27.833068 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.833036    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:27.833137 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.833084    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7" event={"ID":"34451a00-43bd-42a6-af1d-0b9449492697","Type":"ContainerStarted","Data":"5bf64edffd0244993c8bde0fec7d77c528a69da227392a2650e66154c596bc3f"}
Sep 18 21:37:27.861832 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:27.861787    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7" podStartSLOduration=1.398742092 podStartE2EDuration="1.861772004s" totalImagesPullingTime="463.029912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:37:27.861120803 +0000 UTC m=+1885.468180362" watchObservedRunningTime="2026-09-18 21:37:27.861772004 +0000 UTC m=+1885.468831562"
Sep 18 21:37:28.094583 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:28.093138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"]
Sep 18 21:37:28.098218 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:37:28.098178    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99281f53_e477_4f7d_a616_95589bb5da49.slice/crio-174359bc85440b0ed036da21f38fcebe9d3bcb0d3ace22784ada281c73641008 WatchSource:0}: Error finding container 174359bc85440b0ed036da21f38fcebe9d3bcb0d3ace22784ada281c73641008: Status 404 returned error can't find the container with id 174359bc85440b0ed036da21f38fcebe9d3bcb0d3ace22784ada281c73641008
Sep 18 21:37:28.840407 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:28.840364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64" event={"ID":"99281f53-e477-4f7d-a616-95589bb5da49","Type":"ContainerStarted","Data":"d3aed4d79d6e6c2da9dd10dc8e115036fd62a2d505639981bba9ce7b00baea01"}
Sep 18 21:37:28.840843 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:28.840441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"
Sep 18 21:37:28.840843 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:28.840468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64" event={"ID":"99281f53-e477-4f7d-a616-95589bb5da49","Type":"ContainerStarted","Data":"174359bc85440b0ed036da21f38fcebe9d3bcb0d3ace22784ada281c73641008"}
Sep 18 21:37:33.846436 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:33.846404    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:37:33.876060 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:33.876006    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64" podStartSLOduration=6.875991667 podStartE2EDuration="6.875991667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:37:28.879203302 +0000 UTC m=+1886.486262859" watchObservedRunningTime="2026-09-18 21:37:33.875991667 +0000 UTC m=+1891.483051225"
Sep 18 21:37:39.848360 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:39.848332    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"
Sep 18 21:37:49.812504 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:49.812467    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:50.852818 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:50.852785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:51.611726 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:51.611687    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:52.465933 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:52.465893    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:53.278596 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:53.278554    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:53.401166 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:53.401135    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:54.334267 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:54.334237    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:55.121579 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:55.121540    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:55.955606 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:55.955571    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:56.759395 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:56.759359    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:57.718661 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:57.718619    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:58.409966 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:58.409933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:37:59.206533 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:37:59.206500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:02.679961 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:02.679915    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:04.607506 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:04.607451    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:05.221048 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.221010    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"]
Sep 18 21:38:05.221579 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.221521    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7" podUID="34451a00-43bd-42a6-af1d-0b9449492697" containerName="maas-api" containerID="cri-o://faa2fad3dc322b2bb86f5ff0ff9ba88f8bef08c41b8564436e617e04d8eb64e9" gracePeriod=30
Sep 18 21:38:05.251637 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.251606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:05.296820 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.296787    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"]
Sep 18 21:38:05.297073 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.297044    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64" podUID="99281f53-e477-4f7d-a616-95589bb5da49" containerName="payload-pre-processing" containerID="cri-o://d3aed4d79d6e6c2da9dd10dc8e115036fd62a2d505639981bba9ce7b00baea01" gracePeriod=30
Sep 18 21:38:05.355465 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.355433    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:05.454095 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.454058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:05.548355 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.548311    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:05.555968 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.555943    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:38:05.584211 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.584192    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"
Sep 18 21:38:05.644112 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.644080    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2chh2\" (UniqueName: \"kubernetes.io/projected/34451a00-43bd-42a6-af1d-0b9449492697-kube-api-access-2chh2\") pod \"34451a00-43bd-42a6-af1d-0b9449492697\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") "
Sep 18 21:38:05.644112 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.644115    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99281f53-e477-4f7d-a616-95589bb5da49-plugins-config-volume\") pod \"99281f53-e477-4f7d-a616-95589bb5da49\" (UID: \"99281f53-e477-4f7d-a616-95589bb5da49\") "
Sep 18 21:38:05.644601 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.644165    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34451a00-43bd-42a6-af1d-0b9449492697-maas-api-tls\") pod \"34451a00-43bd-42a6-af1d-0b9449492697\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") "
Sep 18 21:38:05.644601 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.644232    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34451a00-43bd-42a6-af1d-0b9449492697-openshift-service-ca\") pod \"34451a00-43bd-42a6-af1d-0b9449492697\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") "
Sep 18 21:38:05.644601 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.644272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34451a00-43bd-42a6-af1d-0b9449492697-metrics-certs\") pod \"34451a00-43bd-42a6-af1d-0b9449492697\" (UID: \"34451a00-43bd-42a6-af1d-0b9449492697\") "
Sep 18 21:38:05.644601 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.644383    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxw8x\" (UniqueName: \"kubernetes.io/projected/99281f53-e477-4f7d-a616-95589bb5da49-kube-api-access-pxw8x\") pod \"99281f53-e477-4f7d-a616-95589bb5da49\" (UID: \"99281f53-e477-4f7d-a616-95589bb5da49\") "
Sep 18 21:38:05.644847 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.644591    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99281f53-e477-4f7d-a616-95589bb5da49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99281f53-e477-4f7d-a616-95589bb5da49" (UID: "99281f53-e477-4f7d-a616-95589bb5da49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:38:05.644847 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.644605    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34451a00-43bd-42a6-af1d-0b9449492697-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "34451a00-43bd-42a6-af1d-0b9449492697" (UID: "34451a00-43bd-42a6-af1d-0b9449492697"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:38:05.644847 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.644760    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34451a00-43bd-42a6-af1d-0b9449492697-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:38:05.644847 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.644784    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99281f53-e477-4f7d-a616-95589bb5da49-plugins-config-volume\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:38:05.646580 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.646550    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99281f53-e477-4f7d-a616-95589bb5da49-kube-api-access-pxw8x" (OuterVolumeSpecName: "kube-api-access-pxw8x") pod "99281f53-e477-4f7d-a616-95589bb5da49" (UID: "99281f53-e477-4f7d-a616-95589bb5da49"). InnerVolumeSpecName "kube-api-access-pxw8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:38:05.646838 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.646811    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34451a00-43bd-42a6-af1d-0b9449492697-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "34451a00-43bd-42a6-af1d-0b9449492697" (UID: "34451a00-43bd-42a6-af1d-0b9449492697"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:38:05.646958 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.646882    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34451a00-43bd-42a6-af1d-0b9449492697-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "34451a00-43bd-42a6-af1d-0b9449492697" (UID: "34451a00-43bd-42a6-af1d-0b9449492697"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:38:05.647004 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.646976    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34451a00-43bd-42a6-af1d-0b9449492697-kube-api-access-2chh2" (OuterVolumeSpecName: "kube-api-access-2chh2") pod "34451a00-43bd-42a6-af1d-0b9449492697" (UID: "34451a00-43bd-42a6-af1d-0b9449492697"). InnerVolumeSpecName "kube-api-access-2chh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:38:05.745533 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.745429    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxw8x\" (UniqueName: \"kubernetes.io/projected/99281f53-e477-4f7d-a616-95589bb5da49-kube-api-access-pxw8x\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:38:05.745533 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.745460    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2chh2\" (UniqueName: \"kubernetes.io/projected/34451a00-43bd-42a6-af1d-0b9449492697-kube-api-access-2chh2\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:38:05.745533 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.745471    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34451a00-43bd-42a6-af1d-0b9449492697-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:38:05.745533 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.745480    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34451a00-43bd-42a6-af1d-0b9449492697-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:38:05.908096 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:05.908057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:06.035321 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.035289    2579 generic.go:356] "Generic (PLEG): container finished" podID="99281f53-e477-4f7d-a616-95589bb5da49" containerID="d3aed4d79d6e6c2da9dd10dc8e115036fd62a2d505639981bba9ce7b00baea01" exitCode=0
Sep 18 21:38:06.035501 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.035331    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"
Sep 18 21:38:06.035501 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.035383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64" event={"ID":"99281f53-e477-4f7d-a616-95589bb5da49","Type":"ContainerDied","Data":"d3aed4d79d6e6c2da9dd10dc8e115036fd62a2d505639981bba9ce7b00baea01"}
Sep 18 21:38:06.035501 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.035434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64" event={"ID":"99281f53-e477-4f7d-a616-95589bb5da49","Type":"ContainerDied","Data":"174359bc85440b0ed036da21f38fcebe9d3bcb0d3ace22784ada281c73641008"}
Sep 18 21:38:06.035501 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.035465    2579 scope.go:117] "RemoveContainer" containerID="d3aed4d79d6e6c2da9dd10dc8e115036fd62a2d505639981bba9ce7b00baea01"
Sep 18 21:38:06.036925 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.036901    2579 generic.go:356] "Generic (PLEG): container finished" podID="34451a00-43bd-42a6-af1d-0b9449492697" containerID="faa2fad3dc322b2bb86f5ff0ff9ba88f8bef08c41b8564436e617e04d8eb64e9" exitCode=0
Sep 18 21:38:06.037067 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.036929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7" event={"ID":"34451a00-43bd-42a6-af1d-0b9449492697","Type":"ContainerDied","Data":"faa2fad3dc322b2bb86f5ff0ff9ba88f8bef08c41b8564436e617e04d8eb64e9"}
Sep 18 21:38:06.037067 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.036950    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"
Sep 18 21:38:06.037067 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.036985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7" event={"ID":"34451a00-43bd-42a6-af1d-0b9449492697","Type":"ContainerDied","Data":"5bf64edffd0244993c8bde0fec7d77c528a69da227392a2650e66154c596bc3f"}
Sep 18 21:38:06.044942 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.044923    2579 scope.go:117] "RemoveContainer" containerID="d3aed4d79d6e6c2da9dd10dc8e115036fd62a2d505639981bba9ce7b00baea01"
Sep 18 21:38:06.045287 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:38:06.045262    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3aed4d79d6e6c2da9dd10dc8e115036fd62a2d505639981bba9ce7b00baea01\": container with ID starting with d3aed4d79d6e6c2da9dd10dc8e115036fd62a2d505639981bba9ce7b00baea01 not found: ID does not exist" containerID="d3aed4d79d6e6c2da9dd10dc8e115036fd62a2d505639981bba9ce7b00baea01"
Sep 18 21:38:06.045396 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.045294    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3aed4d79d6e6c2da9dd10dc8e115036fd62a2d505639981bba9ce7b00baea01"} err="failed to get container status \"d3aed4d79d6e6c2da9dd10dc8e115036fd62a2d505639981bba9ce7b00baea01\": rpc error: code = NotFound desc = could not find container \"d3aed4d79d6e6c2da9dd10dc8e115036fd62a2d505639981bba9ce7b00baea01\": container with ID starting with d3aed4d79d6e6c2da9dd10dc8e115036fd62a2d505639981bba9ce7b00baea01 not found: ID does not exist"
Sep 18 21:38:06.045396 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.045313    2579 scope.go:117] "RemoveContainer" containerID="faa2fad3dc322b2bb86f5ff0ff9ba88f8bef08c41b8564436e617e04d8eb64e9"
Sep 18 21:38:06.054336 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.054318    2579 scope.go:117] "RemoveContainer" containerID="faa2fad3dc322b2bb86f5ff0ff9ba88f8bef08c41b8564436e617e04d8eb64e9"
Sep 18 21:38:06.054616 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:38:06.054597    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa2fad3dc322b2bb86f5ff0ff9ba88f8bef08c41b8564436e617e04d8eb64e9\": container with ID starting with faa2fad3dc322b2bb86f5ff0ff9ba88f8bef08c41b8564436e617e04d8eb64e9 not found: ID does not exist" containerID="faa2fad3dc322b2bb86f5ff0ff9ba88f8bef08c41b8564436e617e04d8eb64e9"
Sep 18 21:38:06.054697 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.054623    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa2fad3dc322b2bb86f5ff0ff9ba88f8bef08c41b8564436e617e04d8eb64e9"} err="failed to get container status \"faa2fad3dc322b2bb86f5ff0ff9ba88f8bef08c41b8564436e617e04d8eb64e9\": rpc error: code = NotFound desc = could not find container \"faa2fad3dc322b2bb86f5ff0ff9ba88f8bef08c41b8564436e617e04d8eb64e9\": container with ID starting with faa2fad3dc322b2bb86f5ff0ff9ba88f8bef08c41b8564436e617e04d8eb64e9 not found: ID does not exist"
Sep 18 21:38:06.084152 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.084128    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"]
Sep 18 21:38:06.100326 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.100300    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b3d150-5969b8c46b-jp9r7"]
Sep 18 21:38:06.122819 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.122786    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"]
Sep 18 21:38:06.139801 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.139774    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b3d150-84d797d7d-lvt64"]
Sep 18 21:38:06.151023 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.150995    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:06.881369 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.881334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:06.923090 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.923059    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34451a00-43bd-42a6-af1d-0b9449492697" path="/var/lib/kubelet/pods/34451a00-43bd-42a6-af1d-0b9449492697/volumes"
Sep 18 21:38:06.923455 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:06.923441    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99281f53-e477-4f7d-a616-95589bb5da49" path="/var/lib/kubelet/pods/99281f53-e477-4f7d-a616-95589bb5da49/volumes"
Sep 18 21:38:07.801793 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:07.801759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:08.648538 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:08.648502    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:09.650851 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:09.650812    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:12.444855 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:12.444824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:12.558013 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:12.557983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:12.786319 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:12.786285    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:16.749083 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:16.749044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:18.202433 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:18.202398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:19.205526 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:19.205491    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:20.560510 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:20.560472    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:21.888475 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:21.888439    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:23.111272 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:23.111235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:32.383270 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.383234    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"]
Sep 18 21:38:32.383711 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.383482    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9" podUID="908b4416-b80f-4ed4-9591-e550ebc15658" containerName="maas-api" containerID="cri-o://74a39bf137c4a190fa1230e11f55c4f5bc597a0116441934612697b26e02257b" gracePeriod=30
Sep 18 21:38:32.472802 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.472768    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:32.638691 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.638623    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:38:32.670315 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.670272    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:32.805307 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.805274    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-metrics-certs\") pod \"908b4416-b80f-4ed4-9591-e550ebc15658\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") "
Sep 18 21:38:32.805520 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.805348    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/908b4416-b80f-4ed4-9591-e550ebc15658-openshift-service-ca\") pod \"908b4416-b80f-4ed4-9591-e550ebc15658\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") "
Sep 18 21:38:32.805520 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.805382    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dw9r\" (UniqueName: \"kubernetes.io/projected/908b4416-b80f-4ed4-9591-e550ebc15658-kube-api-access-4dw9r\") pod \"908b4416-b80f-4ed4-9591-e550ebc15658\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") "
Sep 18 21:38:32.805520 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.805430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-maas-api-tls\") pod \"908b4416-b80f-4ed4-9591-e550ebc15658\" (UID: \"908b4416-b80f-4ed4-9591-e550ebc15658\") "
Sep 18 21:38:32.805764 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.805741    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908b4416-b80f-4ed4-9591-e550ebc15658-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "908b4416-b80f-4ed4-9591-e550ebc15658" (UID: "908b4416-b80f-4ed4-9591-e550ebc15658"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:38:32.807429 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.807399    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908b4416-b80f-4ed4-9591-e550ebc15658-kube-api-access-4dw9r" (OuterVolumeSpecName: "kube-api-access-4dw9r") pod "908b4416-b80f-4ed4-9591-e550ebc15658" (UID: "908b4416-b80f-4ed4-9591-e550ebc15658"). InnerVolumeSpecName "kube-api-access-4dw9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:38:32.807429 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.807419    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "908b4416-b80f-4ed4-9591-e550ebc15658" (UID: "908b4416-b80f-4ed4-9591-e550ebc15658"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:38:32.807573 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.807434    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "908b4416-b80f-4ed4-9591-e550ebc15658" (UID: "908b4416-b80f-4ed4-9591-e550ebc15658"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:38:32.825453 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.825428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:32.906910 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.906820    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:38:32.906910 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.906858    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/908b4416-b80f-4ed4-9591-e550ebc15658-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:38:32.906910 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.906875    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dw9r\" (UniqueName: \"kubernetes.io/projected/908b4416-b80f-4ed4-9591-e550ebc15658-kube-api-access-4dw9r\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:38:32.906910 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.906889    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/908b4416-b80f-4ed4-9591-e550ebc15658-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:38:32.930323 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:32.930283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:33.035423 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:33.035388    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:33.179223 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:33.179130    2579 generic.go:356] "Generic (PLEG): container finished" podID="908b4416-b80f-4ed4-9591-e550ebc15658" containerID="74a39bf137c4a190fa1230e11f55c4f5bc597a0116441934612697b26e02257b" exitCode=0
Sep 18 21:38:33.179223 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:33.179180    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9" event={"ID":"908b4416-b80f-4ed4-9591-e550ebc15658","Type":"ContainerDied","Data":"74a39bf137c4a190fa1230e11f55c4f5bc597a0116441934612697b26e02257b"}
Sep 18 21:38:33.179419 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:33.179228    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9" event={"ID":"908b4416-b80f-4ed4-9591-e550ebc15658","Type":"ContainerDied","Data":"097b0bf4fe00702e478b4d5182eb4c0f44daa5cd5b62ecf18465fed580b789a1"}
Sep 18 21:38:33.179419 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:33.179253    2579 scope.go:117] "RemoveContainer" containerID="74a39bf137c4a190fa1230e11f55c4f5bc597a0116441934612697b26e02257b"
Sep 18 21:38:33.179419 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:33.179187    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"
Sep 18 21:38:33.189300 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:33.189274    2579 scope.go:117] "RemoveContainer" containerID="74a39bf137c4a190fa1230e11f55c4f5bc597a0116441934612697b26e02257b"
Sep 18 21:38:33.189616 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:38:33.189585    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a39bf137c4a190fa1230e11f55c4f5bc597a0116441934612697b26e02257b\": container with ID starting with 74a39bf137c4a190fa1230e11f55c4f5bc597a0116441934612697b26e02257b not found: ID does not exist" containerID="74a39bf137c4a190fa1230e11f55c4f5bc597a0116441934612697b26e02257b"
Sep 18 21:38:33.189705 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:33.189628    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a39bf137c4a190fa1230e11f55c4f5bc597a0116441934612697b26e02257b"} err="failed to get container status \"74a39bf137c4a190fa1230e11f55c4f5bc597a0116441934612697b26e02257b\": rpc error: code = NotFound desc = could not find container \"74a39bf137c4a190fa1230e11f55c4f5bc597a0116441934612697b26e02257b\": container with ID starting with 74a39bf137c4a190fa1230e11f55c4f5bc597a0116441934612697b26e02257b not found: ID does not exist"
Sep 18 21:38:33.220735 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:33.220703    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"]
Sep 18 21:38:33.226275 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:33.226244    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5888e6-5fb97fdf76-7ptg9"]
Sep 18 21:38:33.347498 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:33.347459    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:33.479825 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:33.479731    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:34.867903 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:34.867861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:34.921974 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:34.921939    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908b4416-b80f-4ed4-9591-e550ebc15658" path="/var/lib/kubelet/pods/908b4416-b80f-4ed4-9591-e550ebc15658/volumes"
Sep 18 21:38:35.687744 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:35.687713    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:36.573654 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:36.573604    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:38:38.410676 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:38:38.410626    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:15.194894 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:15.194855    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:16.082893 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:16.082853    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:22.761429 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:22.761392    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:22.878159 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:22.878122    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:23.103078 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:23.103037    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:23.712718 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:23.712678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:24.734443 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:24.734410    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:25.830774 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:25.830735    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:26.756036 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:26.755978    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:27.756108 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:27.756071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:34.105035 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.105001    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"]
Sep 18 21:39:34.105540 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.105230    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp" podUID="ffe5c426-17e6-4e28-9986-c2bc89b18587" containerName="maas-api" containerID="cri-o://8469f1e0429c5ba5ef40cb5d7e6af81bbb8160fe663e7b75d3a2e29afdaddba9" gracePeriod=30
Sep 18 21:39:34.143530 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.143500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:34.239086 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.239055    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:34.347104 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.347076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:34.355968 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.355909    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:39:34.379719 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.379686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffe5c426-17e6-4e28-9986-c2bc89b18587-maas-api-tls\") pod \"ffe5c426-17e6-4e28-9986-c2bc89b18587\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") "
Sep 18 21:39:34.379842 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.379755    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe5c426-17e6-4e28-9986-c2bc89b18587-openshift-service-ca\") pod \"ffe5c426-17e6-4e28-9986-c2bc89b18587\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") "
Sep 18 21:39:34.379960 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.379943    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffe5c426-17e6-4e28-9986-c2bc89b18587-metrics-certs\") pod \"ffe5c426-17e6-4e28-9986-c2bc89b18587\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") "
Sep 18 21:39:34.380005 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.379983    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpvzl\" (UniqueName: \"kubernetes.io/projected/ffe5c426-17e6-4e28-9986-c2bc89b18587-kube-api-access-kpvzl\") pod \"ffe5c426-17e6-4e28-9986-c2bc89b18587\" (UID: \"ffe5c426-17e6-4e28-9986-c2bc89b18587\") "
Sep 18 21:39:34.380104 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.380079    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe5c426-17e6-4e28-9986-c2bc89b18587-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ffe5c426-17e6-4e28-9986-c2bc89b18587" (UID: "ffe5c426-17e6-4e28-9986-c2bc89b18587"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:39:34.380291 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.380273    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe5c426-17e6-4e28-9986-c2bc89b18587-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:39:34.382172 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.382144    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe5c426-17e6-4e28-9986-c2bc89b18587-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ffe5c426-17e6-4e28-9986-c2bc89b18587" (UID: "ffe5c426-17e6-4e28-9986-c2bc89b18587"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:39:34.382545 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.382285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe5c426-17e6-4e28-9986-c2bc89b18587-kube-api-access-kpvzl" (OuterVolumeSpecName: "kube-api-access-kpvzl") pod "ffe5c426-17e6-4e28-9986-c2bc89b18587" (UID: "ffe5c426-17e6-4e28-9986-c2bc89b18587"). InnerVolumeSpecName "kube-api-access-kpvzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:39:34.382545 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.382442    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe5c426-17e6-4e28-9986-c2bc89b18587-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ffe5c426-17e6-4e28-9986-c2bc89b18587" (UID: "ffe5c426-17e6-4e28-9986-c2bc89b18587"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:39:34.453410 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.453373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:34.481514 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.481483    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffe5c426-17e6-4e28-9986-c2bc89b18587-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:39:34.481514 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.481511    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpvzl\" (UniqueName: \"kubernetes.io/projected/ffe5c426-17e6-4e28-9986-c2bc89b18587-kube-api-access-kpvzl\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:39:34.481514 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.481520    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffe5c426-17e6-4e28-9986-c2bc89b18587-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:39:34.483211 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.483180    2579 generic.go:356] "Generic (PLEG): container finished" podID="ffe5c426-17e6-4e28-9986-c2bc89b18587" containerID="8469f1e0429c5ba5ef40cb5d7e6af81bbb8160fe663e7b75d3a2e29afdaddba9" exitCode=0
Sep 18 21:39:34.483349 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.483222    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"
Sep 18 21:39:34.483349 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.483272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp" event={"ID":"ffe5c426-17e6-4e28-9986-c2bc89b18587","Type":"ContainerDied","Data":"8469f1e0429c5ba5ef40cb5d7e6af81bbb8160fe663e7b75d3a2e29afdaddba9"}
Sep 18 21:39:34.483349 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.483331    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp" event={"ID":"ffe5c426-17e6-4e28-9986-c2bc89b18587","Type":"ContainerDied","Data":"00a1c32bb5ccd36555c3b1307ef4dc16fc132d118d99c7b6922994dc0e66532e"}
Sep 18 21:39:34.483491 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.483364    2579 scope.go:117] "RemoveContainer" containerID="8469f1e0429c5ba5ef40cb5d7e6af81bbb8160fe663e7b75d3a2e29afdaddba9"
Sep 18 21:39:34.492906 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.492885    2579 scope.go:117] "RemoveContainer" containerID="8469f1e0429c5ba5ef40cb5d7e6af81bbb8160fe663e7b75d3a2e29afdaddba9"
Sep 18 21:39:34.493195 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:39:34.493173    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8469f1e0429c5ba5ef40cb5d7e6af81bbb8160fe663e7b75d3a2e29afdaddba9\": container with ID starting with 8469f1e0429c5ba5ef40cb5d7e6af81bbb8160fe663e7b75d3a2e29afdaddba9 not found: ID does not exist" containerID="8469f1e0429c5ba5ef40cb5d7e6af81bbb8160fe663e7b75d3a2e29afdaddba9"
Sep 18 21:39:34.493295 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.493204    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8469f1e0429c5ba5ef40cb5d7e6af81bbb8160fe663e7b75d3a2e29afdaddba9"} err="failed to get container status \"8469f1e0429c5ba5ef40cb5d7e6af81bbb8160fe663e7b75d3a2e29afdaddba9\": rpc error: code = NotFound desc = could not find container \"8469f1e0429c5ba5ef40cb5d7e6af81bbb8160fe663e7b75d3a2e29afdaddba9\": container with ID starting with 8469f1e0429c5ba5ef40cb5d7e6af81bbb8160fe663e7b75d3a2e29afdaddba9 not found: ID does not exist"
Sep 18 21:39:34.517788 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.517759    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"]
Sep 18 21:39:34.523158 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.523131    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-deac7a-6959d8cf5d-gsbzp"]
Sep 18 21:39:34.895593 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.895561    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:34.921768 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:34.921736    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffe5c426-17e6-4e28-9986-c2bc89b18587" path="/var/lib/kubelet/pods/ffe5c426-17e6-4e28-9986-c2bc89b18587/volumes"
Sep 18 21:39:35.218234 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:35.218152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:36.227189 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:36.227160    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:37.711215 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:37.711179    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:42.310193 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:42.310158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:42.407762 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:42.407725    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:42.632335 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:42.632251    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:43.270832 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:43.270778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:44.344264 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:44.344233    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:45.366839 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:45.366801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:46.356915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:46.356886    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:53.672535 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:53.672500    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"]
Sep 18 21:39:53.672939 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:53.672743    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl" podUID="7635a1a0-9bd8-483d-a59e-c89459fbf72a" containerName="maas-api" containerID="cri-o://3ed87ecfc5984c3de0c65163d3ab918537694f1898c8e32a5d1d4e528ba6f9ec" gracePeriod=30
Sep 18 21:39:53.698153 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:53.698100    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:53.799405 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:53.799375    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:53.911451 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:53.911429    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:53.920696 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:53.920672    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:39:54.009012 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.008931    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:54.083324 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.083292    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-maas-api-tls\") pod \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") "
Sep 18 21:39:54.083495 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.083338    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7635a1a0-9bd8-483d-a59e-c89459fbf72a-openshift-service-ca\") pod \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") "
Sep 18 21:39:54.083495 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.083378    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st2dk\" (UniqueName: \"kubernetes.io/projected/7635a1a0-9bd8-483d-a59e-c89459fbf72a-kube-api-access-st2dk\") pod \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") "
Sep 18 21:39:54.083495 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.083426    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-metrics-certs\") pod \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\" (UID: \"7635a1a0-9bd8-483d-a59e-c89459fbf72a\") "
Sep 18 21:39:54.083808 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.083780    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7635a1a0-9bd8-483d-a59e-c89459fbf72a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7635a1a0-9bd8-483d-a59e-c89459fbf72a" (UID: "7635a1a0-9bd8-483d-a59e-c89459fbf72a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:39:54.085606 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.085570    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7635a1a0-9bd8-483d-a59e-c89459fbf72a" (UID: "7635a1a0-9bd8-483d-a59e-c89459fbf72a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:39:54.085606 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.085584    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7635a1a0-9bd8-483d-a59e-c89459fbf72a-kube-api-access-st2dk" (OuterVolumeSpecName: "kube-api-access-st2dk") pod "7635a1a0-9bd8-483d-a59e-c89459fbf72a" (UID: "7635a1a0-9bd8-483d-a59e-c89459fbf72a"). InnerVolumeSpecName "kube-api-access-st2dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:39:54.085771 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.085614    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7635a1a0-9bd8-483d-a59e-c89459fbf72a" (UID: "7635a1a0-9bd8-483d-a59e-c89459fbf72a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:39:54.184452 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.184415    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:39:54.184452 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.184447    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7635a1a0-9bd8-483d-a59e-c89459fbf72a-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:39:54.184452 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.184457    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st2dk\" (UniqueName: \"kubernetes.io/projected/7635a1a0-9bd8-483d-a59e-c89459fbf72a-kube-api-access-st2dk\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:39:54.184726 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.184465    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7635a1a0-9bd8-483d-a59e-c89459fbf72a-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:39:54.403915 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.403881    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:54.508652 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.508617    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:39:54.586418 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.586384    2579 generic.go:356] "Generic (PLEG): container finished" podID="7635a1a0-9bd8-483d-a59e-c89459fbf72a" containerID="3ed87ecfc5984c3de0c65163d3ab918537694f1898c8e32a5d1d4e528ba6f9ec" exitCode=0
Sep 18 21:39:54.586602 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.586427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl" event={"ID":"7635a1a0-9bd8-483d-a59e-c89459fbf72a","Type":"ContainerDied","Data":"3ed87ecfc5984c3de0c65163d3ab918537694f1898c8e32a5d1d4e528ba6f9ec"}
Sep 18 21:39:54.586602 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.586437    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"
Sep 18 21:39:54.586602 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.586471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl" event={"ID":"7635a1a0-9bd8-483d-a59e-c89459fbf72a","Type":"ContainerDied","Data":"b45166a922fccba71f49245ff951d3f59eb6a318c032e9bbf1010f0636124db8"}
Sep 18 21:39:54.586602 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.586492    2579 scope.go:117] "RemoveContainer" containerID="3ed87ecfc5984c3de0c65163d3ab918537694f1898c8e32a5d1d4e528ba6f9ec"
Sep 18 21:39:54.595975 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.595954    2579 scope.go:117] "RemoveContainer" containerID="3ed87ecfc5984c3de0c65163d3ab918537694f1898c8e32a5d1d4e528ba6f9ec"
Sep 18 21:39:54.596259 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:39:54.596240    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed87ecfc5984c3de0c65163d3ab918537694f1898c8e32a5d1d4e528ba6f9ec\": container with ID starting with 3ed87ecfc5984c3de0c65163d3ab918537694f1898c8e32a5d1d4e528ba6f9ec not found: ID does not exist" containerID="3ed87ecfc5984c3de0c65163d3ab918537694f1898c8e32a5d1d4e528ba6f9ec"
Sep 18 21:39:54.596324 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.596270    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed87ecfc5984c3de0c65163d3ab918537694f1898c8e32a5d1d4e528ba6f9ec"} err="failed to get container status \"3ed87ecfc5984c3de0c65163d3ab918537694f1898c8e32a5d1d4e528ba6f9ec\": rpc error: code = NotFound desc = could not find container \"3ed87ecfc5984c3de0c65163d3ab918537694f1898c8e32a5d1d4e528ba6f9ec\": container with ID starting with 3ed87ecfc5984c3de0c65163d3ab918537694f1898c8e32a5d1d4e528ba6f9ec not found: ID does not exist"
Sep 18 21:39:54.609375 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.609351    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"]
Sep 18 21:39:54.614947 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.614925    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cad5cf-744fbb86df-76spl"]
Sep 18 21:39:54.921776 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:54.921738    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7635a1a0-9bd8-483d-a59e-c89459fbf72a" path="/var/lib/kubelet/pods/7635a1a0-9bd8-483d-a59e-c89459fbf72a/volumes"
Sep 18 21:39:55.330109 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:39:55.330078    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:01.302195 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:01.302160    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:01.403828 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:01.403796    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:01.604900 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:01.604813    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:03.018555 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:03.018521    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:04.008846 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:04.008805    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:08.450725 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:08.450687    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:12.695977 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:12.695937    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"]
Sep 18 21:40:12.696422 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:12.696190    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m" podUID="6a60318f-3709-4820-890e-0722dda8faaa" containerName="maas-api" containerID="cri-o://d7359a5aed830656815c4ac76b531c3a85b493a6588036ff0041feb5f5bdfc44" gracePeriod=30
Sep 18 21:40:12.782205 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:12.782174    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:12.902862 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:12.902829    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:12.957389 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:12.957328    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:40:13.018513 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.018482    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:13.064335 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.064297    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shg5t\" (UniqueName: \"kubernetes.io/projected/6a60318f-3709-4820-890e-0722dda8faaa-kube-api-access-shg5t\") pod \"6a60318f-3709-4820-890e-0722dda8faaa\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") "
Sep 18 21:40:13.064530 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.064396    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a60318f-3709-4820-890e-0722dda8faaa-openshift-service-ca\") pod \"6a60318f-3709-4820-890e-0722dda8faaa\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") "
Sep 18 21:40:13.064530 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.064452    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a60318f-3709-4820-890e-0722dda8faaa-maas-api-tls\") pod \"6a60318f-3709-4820-890e-0722dda8faaa\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") "
Sep 18 21:40:13.064530 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.064512    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a60318f-3709-4820-890e-0722dda8faaa-metrics-certs\") pod \"6a60318f-3709-4820-890e-0722dda8faaa\" (UID: \"6a60318f-3709-4820-890e-0722dda8faaa\") "
Sep 18 21:40:13.064825 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.064796    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a60318f-3709-4820-890e-0722dda8faaa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a60318f-3709-4820-890e-0722dda8faaa" (UID: "6a60318f-3709-4820-890e-0722dda8faaa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:40:13.066686 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.066628    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a60318f-3709-4820-890e-0722dda8faaa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6a60318f-3709-4820-890e-0722dda8faaa" (UID: "6a60318f-3709-4820-890e-0722dda8faaa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:40:13.066933 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.066906    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a60318f-3709-4820-890e-0722dda8faaa-kube-api-access-shg5t" (OuterVolumeSpecName: "kube-api-access-shg5t") pod "6a60318f-3709-4820-890e-0722dda8faaa" (UID: "6a60318f-3709-4820-890e-0722dda8faaa"). InnerVolumeSpecName "kube-api-access-shg5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:40:13.067034 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.066916    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a60318f-3709-4820-890e-0722dda8faaa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a60318f-3709-4820-890e-0722dda8faaa" (UID: "6a60318f-3709-4820-890e-0722dda8faaa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:40:13.123271 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.123228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:13.166375 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.166340    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a60318f-3709-4820-890e-0722dda8faaa-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:40:13.166375 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.166372    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a60318f-3709-4820-890e-0722dda8faaa-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:40:13.166589 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.166385    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a60318f-3709-4820-890e-0722dda8faaa-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:40:13.166589 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.166399    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shg5t\" (UniqueName: \"kubernetes.io/projected/6a60318f-3709-4820-890e-0722dda8faaa-kube-api-access-shg5t\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:40:13.479170 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.479142    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:13.578496 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.578461    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:13.678691 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.678626    2579 generic.go:356] "Generic (PLEG): container finished" podID="6a60318f-3709-4820-890e-0722dda8faaa" containerID="d7359a5aed830656815c4ac76b531c3a85b493a6588036ff0041feb5f5bdfc44" exitCode=0
Sep 18 21:40:13.678691 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.678687    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"
Sep 18 21:40:13.678961 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.678714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m" event={"ID":"6a60318f-3709-4820-890e-0722dda8faaa","Type":"ContainerDied","Data":"d7359a5aed830656815c4ac76b531c3a85b493a6588036ff0041feb5f5bdfc44"}
Sep 18 21:40:13.678961 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.678778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m" event={"ID":"6a60318f-3709-4820-890e-0722dda8faaa","Type":"ContainerDied","Data":"93ca1046419408f902473c9bc2a04b1d55821a71a5d340ba2c9f33b0814dee67"}
Sep 18 21:40:13.678961 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.678807    2579 scope.go:117] "RemoveContainer" containerID="d7359a5aed830656815c4ac76b531c3a85b493a6588036ff0041feb5f5bdfc44"
Sep 18 21:40:13.688152 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.688136    2579 scope.go:117] "RemoveContainer" containerID="d7359a5aed830656815c4ac76b531c3a85b493a6588036ff0041feb5f5bdfc44"
Sep 18 21:40:13.688416 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:40:13.688396    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7359a5aed830656815c4ac76b531c3a85b493a6588036ff0041feb5f5bdfc44\": container with ID starting with d7359a5aed830656815c4ac76b531c3a85b493a6588036ff0041feb5f5bdfc44 not found: ID does not exist" containerID="d7359a5aed830656815c4ac76b531c3a85b493a6588036ff0041feb5f5bdfc44"
Sep 18 21:40:13.688486 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.688429    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7359a5aed830656815c4ac76b531c3a85b493a6588036ff0041feb5f5bdfc44"} err="failed to get container status \"d7359a5aed830656815c4ac76b531c3a85b493a6588036ff0041feb5f5bdfc44\": rpc error: code = NotFound desc = could not find container \"d7359a5aed830656815c4ac76b531c3a85b493a6588036ff0041feb5f5bdfc44\": container with ID starting with d7359a5aed830656815c4ac76b531c3a85b493a6588036ff0041feb5f5bdfc44 not found: ID does not exist"
Sep 18 21:40:13.707756 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.707729    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"]
Sep 18 21:40:13.711334 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:13.711315    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fd993e-6ccdb75b8-7bx2m"]
Sep 18 21:40:14.921784 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:14.921739    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a60318f-3709-4820-890e-0722dda8faaa" path="/var/lib/kubelet/pods/6a60318f-3709-4820-890e-0722dda8faaa/volumes"
Sep 18 21:40:16.716591 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:16.716533    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:16.850554 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:16.850520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:21.363985 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:21.363932    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:21.488703 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:21.488671    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:21.811318 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:21.811270    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:40:24.457859 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.457820    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-799c76f65c-zjvqk"]
Sep 18 21:40:24.458286 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.458097    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-799c76f65c-zjvqk" podUID="806deb32-abef-4295-9a73-4693b2682de2" containerName="manager" containerID="cri-o://45613ec5d9805b3380554ac1c5ed768219f434df65f9714e69e83030916404db" gracePeriod=30
Sep 18 21:40:24.705254 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.705228    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:40:24.739324 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.739247    2579 generic.go:356] "Generic (PLEG): container finished" podID="806deb32-abef-4295-9a73-4693b2682de2" containerID="45613ec5d9805b3380554ac1c5ed768219f434df65f9714e69e83030916404db" exitCode=0
Sep 18 21:40:24.739570 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.739320    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799c76f65c-zjvqk"
Sep 18 21:40:24.739750 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.739361    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799c76f65c-zjvqk" event={"ID":"806deb32-abef-4295-9a73-4693b2682de2","Type":"ContainerDied","Data":"45613ec5d9805b3380554ac1c5ed768219f434df65f9714e69e83030916404db"}
Sep 18 21:40:24.739944 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.739915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799c76f65c-zjvqk" event={"ID":"806deb32-abef-4295-9a73-4693b2682de2","Type":"ContainerDied","Data":"7b7ba91299f9fcec8e16877492bfcfe92ae3f7d41fe523a3639342b8eb185e9a"}
Sep 18 21:40:24.740070 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.739962    2579 scope.go:117] "RemoveContainer" containerID="45613ec5d9805b3380554ac1c5ed768219f434df65f9714e69e83030916404db"
Sep 18 21:40:24.751567 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.751547    2579 scope.go:117] "RemoveContainer" containerID="45613ec5d9805b3380554ac1c5ed768219f434df65f9714e69e83030916404db"
Sep 18 21:40:24.751921 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:40:24.751894    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45613ec5d9805b3380554ac1c5ed768219f434df65f9714e69e83030916404db\": container with ID starting with 45613ec5d9805b3380554ac1c5ed768219f434df65f9714e69e83030916404db not found: ID does not exist" containerID="45613ec5d9805b3380554ac1c5ed768219f434df65f9714e69e83030916404db"
Sep 18 21:40:24.752010 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.751935    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45613ec5d9805b3380554ac1c5ed768219f434df65f9714e69e83030916404db"} err="failed to get container status \"45613ec5d9805b3380554ac1c5ed768219f434df65f9714e69e83030916404db\": rpc error: code = NotFound desc = could not find container \"45613ec5d9805b3380554ac1c5ed768219f434df65f9714e69e83030916404db\": container with ID starting with 45613ec5d9805b3380554ac1c5ed768219f434df65f9714e69e83030916404db not found: ID does not exist"
Sep 18 21:40:24.781262 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.781236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6lhg\" (UniqueName: \"kubernetes.io/projected/806deb32-abef-4295-9a73-4693b2682de2-kube-api-access-c6lhg\") pod \"806deb32-abef-4295-9a73-4693b2682de2\" (UID: \"806deb32-abef-4295-9a73-4693b2682de2\") "
Sep 18 21:40:24.781384 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.781286    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/806deb32-abef-4295-9a73-4693b2682de2-metrics-certs\") pod \"806deb32-abef-4295-9a73-4693b2682de2\" (UID: \"806deb32-abef-4295-9a73-4693b2682de2\") "
Sep 18 21:40:24.781384 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.781324    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/806deb32-abef-4295-9a73-4693b2682de2-webhook-cert\") pod \"806deb32-abef-4295-9a73-4693b2682de2\" (UID: \"806deb32-abef-4295-9a73-4693b2682de2\") "
Sep 18 21:40:24.783348 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.783319    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806deb32-abef-4295-9a73-4693b2682de2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "806deb32-abef-4295-9a73-4693b2682de2" (UID: "806deb32-abef-4295-9a73-4693b2682de2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:40:24.783348 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.783319    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806deb32-abef-4295-9a73-4693b2682de2-kube-api-access-c6lhg" (OuterVolumeSpecName: "kube-api-access-c6lhg") pod "806deb32-abef-4295-9a73-4693b2682de2" (UID: "806deb32-abef-4295-9a73-4693b2682de2"). InnerVolumeSpecName "kube-api-access-c6lhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:40:24.783477 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.783339    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806deb32-abef-4295-9a73-4693b2682de2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "806deb32-abef-4295-9a73-4693b2682de2" (UID: "806deb32-abef-4295-9a73-4693b2682de2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:40:24.882317 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.882264    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6lhg\" (UniqueName: \"kubernetes.io/projected/806deb32-abef-4295-9a73-4693b2682de2-kube-api-access-c6lhg\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:40:24.882317 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.882311    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/806deb32-abef-4295-9a73-4693b2682de2-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:40:24.882317 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:24.882325    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/806deb32-abef-4295-9a73-4693b2682de2-webhook-cert\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:40:25.058225 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:25.058185    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-799c76f65c-zjvqk"]
Sep 18 21:40:25.061114 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:25.061087    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-799c76f65c-zjvqk"]
Sep 18 21:40:26.921958 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:40:26.921924    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="806deb32-abef-4295-9a73-4693b2682de2" path="/var/lib/kubelet/pods/806deb32-abef-4295-9a73-4693b2682de2/volumes"
Sep 18 21:41:02.996136 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:02.996103    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:41:02.999721 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:02.999701    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:41:03.003572 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:03.003552    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:41:03.007301 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:03.007285    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:41:25.947230 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947192    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-799c76f65c-rvqdb"]
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947559    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7635a1a0-9bd8-483d-a59e-c89459fbf72a" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947570    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7635a1a0-9bd8-483d-a59e-c89459fbf72a" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947580    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34451a00-43bd-42a6-af1d-0b9449492697" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947586    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34451a00-43bd-42a6-af1d-0b9449492697" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947594    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99281f53-e477-4f7d-a616-95589bb5da49" containerName="payload-pre-processing"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947600    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="99281f53-e477-4f7d-a616-95589bb5da49" containerName="payload-pre-processing"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947616    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908b4416-b80f-4ed4-9591-e550ebc15658" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947621    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="908b4416-b80f-4ed4-9591-e550ebc15658" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947627    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="806deb32-abef-4295-9a73-4693b2682de2" containerName="manager"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947633    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="806deb32-abef-4295-9a73-4693b2682de2" containerName="manager"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947659    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffe5c426-17e6-4e28-9986-c2bc89b18587" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947665    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffe5c426-17e6-4e28-9986-c2bc89b18587" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947676    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a60318f-3709-4820-890e-0722dda8faaa" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947681    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a60318f-3709-4820-890e-0722dda8faaa" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947733    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffe5c426-17e6-4e28-9986-c2bc89b18587" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947742    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="99281f53-e477-4f7d-a616-95589bb5da49" containerName="payload-pre-processing"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947750    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a60318f-3709-4820-890e-0722dda8faaa" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947756    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="908b4416-b80f-4ed4-9591-e550ebc15658" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947762    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7635a1a0-9bd8-483d-a59e-c89459fbf72a" containerName="maas-api"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947768    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="806deb32-abef-4295-9a73-4693b2682de2" containerName="manager"
Sep 18 21:41:25.947765 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.947774    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34451a00-43bd-42a6-af1d-0b9449492697" containerName="maas-api"
Sep 18 21:41:25.950853 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.950836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:25.955458 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.955434    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ghw2c\""
Sep 18 21:41:25.955573 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.955444    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 21:41:25.955573 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.955560    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 21:41:25.955664 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.955625    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 21:41:25.982830 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:25.982798    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-799c76f65c-rvqdb"]
Sep 18 21:41:26.039206 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:26.039173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-webhook-cert\") pod \"maas-controller-799c76f65c-rvqdb\" (UID: \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\") " pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:26.039355 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:26.039301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxljq\" (UniqueName: \"kubernetes.io/projected/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-kube-api-access-vxljq\") pod \"maas-controller-799c76f65c-rvqdb\" (UID: \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\") " pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:26.039401 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:26.039350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-metrics-certs\") pod \"maas-controller-799c76f65c-rvqdb\" (UID: \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\") " pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:26.140076 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:26.140041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxljq\" (UniqueName: \"kubernetes.io/projected/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-kube-api-access-vxljq\") pod \"maas-controller-799c76f65c-rvqdb\" (UID: \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\") " pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:26.140241 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:26.140094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-metrics-certs\") pod \"maas-controller-799c76f65c-rvqdb\" (UID: \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\") " pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:26.140241 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:26.140132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-webhook-cert\") pod \"maas-controller-799c76f65c-rvqdb\" (UID: \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\") " pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:26.142719 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:26.142683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-webhook-cert\") pod \"maas-controller-799c76f65c-rvqdb\" (UID: \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\") " pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:26.142859 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:26.142721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-metrics-certs\") pod \"maas-controller-799c76f65c-rvqdb\" (UID: \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\") " pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:26.149538 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:26.149513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxljq\" (UniqueName: \"kubernetes.io/projected/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-kube-api-access-vxljq\") pod \"maas-controller-799c76f65c-rvqdb\" (UID: \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\") " pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:26.260933 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:26.260845    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:26.547325 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:26.547292    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-799c76f65c-rvqdb"]
Sep 18 21:41:26.551677 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:41:26.551615    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34560ef5_e95e_4dbe_a6bb_4c2395f3d165.slice/crio-04052c4de9cdea41fc7ed54511e8d1cb445217559ddd76c21a4cba015d8433fa WatchSource:0}: Error finding container 04052c4de9cdea41fc7ed54511e8d1cb445217559ddd76c21a4cba015d8433fa: Status 404 returned error can't find the container with id 04052c4de9cdea41fc7ed54511e8d1cb445217559ddd76c21a4cba015d8433fa
Sep 18 21:41:26.553317 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:26.553300    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:41:27.049838 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:27.049811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799c76f65c-rvqdb" event={"ID":"34560ef5-e95e-4dbe-a6bb-4c2395f3d165","Type":"ContainerStarted","Data":"beced5a61cc0861683ff04f29f57db207f66a65b9979ebc3bc5570892940328a"}
Sep 18 21:41:27.050142 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:27.049859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799c76f65c-rvqdb" event={"ID":"34560ef5-e95e-4dbe-a6bb-4c2395f3d165","Type":"ContainerStarted","Data":"04052c4de9cdea41fc7ed54511e8d1cb445217559ddd76c21a4cba015d8433fa"}
Sep 18 21:41:27.050142 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:27.049891    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:27.070625 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:27.070579    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-799c76f65c-rvqdb" podStartSLOduration=1.6365476650000002 podStartE2EDuration="2.070562841s" totalImagesPullingTime="434.015176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:41:27.07050315 +0000 UTC m=+2124.677562711" watchObservedRunningTime="2026-09-18 21:41:27.070562841 +0000 UTC m=+2124.677622399"
Sep 18 21:41:38.060174 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:38.060138    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:47.350453 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:47.350399    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:41:47.447389 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:47.447352    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-799c76f65c-rvqdb"]
Sep 18 21:41:47.447600 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:47.447575    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-799c76f65c-rvqdb" podUID="34560ef5-e95e-4dbe-a6bb-4c2395f3d165" containerName="manager" containerID="cri-o://beced5a61cc0861683ff04f29f57db207f66a65b9979ebc3bc5570892940328a" gracePeriod=30
Sep 18 21:41:47.696441 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:47.696415    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:47.746520 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:47.746490    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-metrics-certs\") pod \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\" (UID: \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\") "
Sep 18 21:41:47.746719 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:47.746596    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxljq\" (UniqueName: \"kubernetes.io/projected/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-kube-api-access-vxljq\") pod \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\" (UID: \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\") "
Sep 18 21:41:47.746719 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:47.746637    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-webhook-cert\") pod \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\" (UID: \"34560ef5-e95e-4dbe-a6bb-4c2395f3d165\") "
Sep 18 21:41:47.750343 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:47.750252    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "34560ef5-e95e-4dbe-a6bb-4c2395f3d165" (UID: "34560ef5-e95e-4dbe-a6bb-4c2395f3d165"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:41:47.750343 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:47.750275    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-kube-api-access-vxljq" (OuterVolumeSpecName: "kube-api-access-vxljq") pod "34560ef5-e95e-4dbe-a6bb-4c2395f3d165" (UID: "34560ef5-e95e-4dbe-a6bb-4c2395f3d165"). InnerVolumeSpecName "kube-api-access-vxljq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:41:47.750343 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:47.750327    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "34560ef5-e95e-4dbe-a6bb-4c2395f3d165" (UID: "34560ef5-e95e-4dbe-a6bb-4c2395f3d165"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:41:47.847886 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:47.847853    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxljq\" (UniqueName: \"kubernetes.io/projected/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-kube-api-access-vxljq\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:41:47.847886 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:47.847881    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-webhook-cert\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:41:47.847886 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:47.847890    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34560ef5-e95e-4dbe-a6bb-4c2395f3d165-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:41:48.156877 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:48.156841    2579 generic.go:356] "Generic (PLEG): container finished" podID="34560ef5-e95e-4dbe-a6bb-4c2395f3d165" containerID="beced5a61cc0861683ff04f29f57db207f66a65b9979ebc3bc5570892940328a" exitCode=0
Sep 18 21:41:48.157053 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:48.156900    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799c76f65c-rvqdb" event={"ID":"34560ef5-e95e-4dbe-a6bb-4c2395f3d165","Type":"ContainerDied","Data":"beced5a61cc0861683ff04f29f57db207f66a65b9979ebc3bc5570892940328a"}
Sep 18 21:41:48.157053 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:48.156949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799c76f65c-rvqdb" event={"ID":"34560ef5-e95e-4dbe-a6bb-4c2395f3d165","Type":"ContainerDied","Data":"04052c4de9cdea41fc7ed54511e8d1cb445217559ddd76c21a4cba015d8433fa"}
Sep 18 21:41:48.157053 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:48.156972    2579 scope.go:117] "RemoveContainer" containerID="beced5a61cc0861683ff04f29f57db207f66a65b9979ebc3bc5570892940328a"
Sep 18 21:41:48.157053 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:48.156901    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799c76f65c-rvqdb"
Sep 18 21:41:48.173502 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:48.173480    2579 scope.go:117] "RemoveContainer" containerID="beced5a61cc0861683ff04f29f57db207f66a65b9979ebc3bc5570892940328a"
Sep 18 21:41:48.173815 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:41:48.173793    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beced5a61cc0861683ff04f29f57db207f66a65b9979ebc3bc5570892940328a\": container with ID starting with beced5a61cc0861683ff04f29f57db207f66a65b9979ebc3bc5570892940328a not found: ID does not exist" containerID="beced5a61cc0861683ff04f29f57db207f66a65b9979ebc3bc5570892940328a"
Sep 18 21:41:48.173917 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:48.173822    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beced5a61cc0861683ff04f29f57db207f66a65b9979ebc3bc5570892940328a"} err="failed to get container status \"beced5a61cc0861683ff04f29f57db207f66a65b9979ebc3bc5570892940328a\": rpc error: code = NotFound desc = could not find container \"beced5a61cc0861683ff04f29f57db207f66a65b9979ebc3bc5570892940328a\": container with ID starting with beced5a61cc0861683ff04f29f57db207f66a65b9979ebc3bc5570892940328a not found: ID does not exist"
Sep 18 21:41:48.190605 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:48.190576    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-799c76f65c-rvqdb"]
Sep 18 21:41:48.202091 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:41:48.202063    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34560ef5_e95e_4dbe_a6bb_4c2395f3d165.slice\": RecentStats: unable to find data in memory cache]"
Sep 18 21:41:48.202349 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:48.202329    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-799c76f65c-rvqdb"]
Sep 18 21:41:48.900525 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:48.900486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:41:48.922245 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:48.922217    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34560ef5-e95e-4dbe-a6bb-4c2395f3d165" path="/var/lib/kubelet/pods/34560ef5-e95e-4dbe-a6bb-4c2395f3d165/volumes"
Sep 18 21:41:49.012282 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.012248    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-799c76f65c-h2stq"]
Sep 18 21:41:49.012656 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.012631    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34560ef5-e95e-4dbe-a6bb-4c2395f3d165" containerName="manager"
Sep 18 21:41:49.012656 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.012656    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34560ef5-e95e-4dbe-a6bb-4c2395f3d165" containerName="manager"
Sep 18 21:41:49.012754 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.012721    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34560ef5-e95e-4dbe-a6bb-4c2395f3d165" containerName="manager"
Sep 18 21:41:49.017197 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.017179    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799c76f65c-h2stq"
Sep 18 21:41:49.020382 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.020359    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ghw2c\""
Sep 18 21:41:49.021365 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.021345    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 21:41:49.021479 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.021459    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 21:41:49.021611 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.021595    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 21:41:49.030476 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.030455    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-799c76f65c-h2stq"]
Sep 18 21:41:49.061013 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.060989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9cf727f5-89d5-4fa0-b738-54147cd30f80-webhook-cert\") pod \"maas-controller-799c76f65c-h2stq\" (UID: \"9cf727f5-89d5-4fa0-b738-54147cd30f80\") " pod="opendatahub/maas-controller-799c76f65c-h2stq"
Sep 18 21:41:49.061132 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.061025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf727f5-89d5-4fa0-b738-54147cd30f80-metrics-certs\") pod \"maas-controller-799c76f65c-h2stq\" (UID: \"9cf727f5-89d5-4fa0-b738-54147cd30f80\") " pod="opendatahub/maas-controller-799c76f65c-h2stq"
Sep 18 21:41:49.061132 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.061113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7gbg\" (UniqueName: \"kubernetes.io/projected/9cf727f5-89d5-4fa0-b738-54147cd30f80-kube-api-access-d7gbg\") pod \"maas-controller-799c76f65c-h2stq\" (UID: \"9cf727f5-89d5-4fa0-b738-54147cd30f80\") " pod="opendatahub/maas-controller-799c76f65c-h2stq"
Sep 18 21:41:49.161874 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.161784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9cf727f5-89d5-4fa0-b738-54147cd30f80-webhook-cert\") pod \"maas-controller-799c76f65c-h2stq\" (UID: \"9cf727f5-89d5-4fa0-b738-54147cd30f80\") " pod="opendatahub/maas-controller-799c76f65c-h2stq"
Sep 18 21:41:49.161874 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.161823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf727f5-89d5-4fa0-b738-54147cd30f80-metrics-certs\") pod \"maas-controller-799c76f65c-h2stq\" (UID: \"9cf727f5-89d5-4fa0-b738-54147cd30f80\") " pod="opendatahub/maas-controller-799c76f65c-h2stq"
Sep 18 21:41:49.162094 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.161948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7gbg\" (UniqueName: \"kubernetes.io/projected/9cf727f5-89d5-4fa0-b738-54147cd30f80-kube-api-access-d7gbg\") pod \"maas-controller-799c76f65c-h2stq\" (UID: \"9cf727f5-89d5-4fa0-b738-54147cd30f80\") " pod="opendatahub/maas-controller-799c76f65c-h2stq"
Sep 18 21:41:49.164342 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.164313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9cf727f5-89d5-4fa0-b738-54147cd30f80-webhook-cert\") pod \"maas-controller-799c76f65c-h2stq\" (UID: \"9cf727f5-89d5-4fa0-b738-54147cd30f80\") " pod="opendatahub/maas-controller-799c76f65c-h2stq"
Sep 18 21:41:49.164457 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.164352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf727f5-89d5-4fa0-b738-54147cd30f80-metrics-certs\") pod \"maas-controller-799c76f65c-h2stq\" (UID: \"9cf727f5-89d5-4fa0-b738-54147cd30f80\") " pod="opendatahub/maas-controller-799c76f65c-h2stq"
Sep 18 21:41:49.173228 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.173205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7gbg\" (UniqueName: \"kubernetes.io/projected/9cf727f5-89d5-4fa0-b738-54147cd30f80-kube-api-access-d7gbg\") pod \"maas-controller-799c76f65c-h2stq\" (UID: \"9cf727f5-89d5-4fa0-b738-54147cd30f80\") " pod="opendatahub/maas-controller-799c76f65c-h2stq"
Sep 18 21:41:49.327430 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.327392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-799c76f65c-h2stq"
Sep 18 21:41:49.608354 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:49.608307    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-799c76f65c-h2stq"]
Sep 18 21:41:49.610626 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:41:49.610597    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf727f5_89d5_4fa0_b738_54147cd30f80.slice/crio-328cb93e1d20badc836cb7294f46cf71197bf0248ff4a98fa7e9a5c4adb7f338 WatchSource:0}: Error finding container 328cb93e1d20badc836cb7294f46cf71197bf0248ff4a98fa7e9a5c4adb7f338: Status 404 returned error can't find the container with id 328cb93e1d20badc836cb7294f46cf71197bf0248ff4a98fa7e9a5c4adb7f338
Sep 18 21:41:50.170100 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:50.170065    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799c76f65c-h2stq" event={"ID":"9cf727f5-89d5-4fa0-b738-54147cd30f80","Type":"ContainerStarted","Data":"56effc4f750ac89d68bfdfc5625c45bd48116d790cb034f266c0ed46561937f9"}
Sep 18 21:41:50.170480 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:50.170119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-799c76f65c-h2stq" event={"ID":"9cf727f5-89d5-4fa0-b738-54147cd30f80","Type":"ContainerStarted","Data":"328cb93e1d20badc836cb7294f46cf71197bf0248ff4a98fa7e9a5c4adb7f338"}
Sep 18 21:41:50.170480 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:41:50.170161    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-799c76f65c-h2stq"
Sep 18 21:42:01.177344 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:42:01.177264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-799c76f65c-h2stq"
Sep 18 21:42:01.204334 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:42:01.204280    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-799c76f65c-h2stq" podStartSLOduration=12.832239138 podStartE2EDuration="13.204264285s" totalImagesPullingTime="372.025147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:41:50.203319738 +0000 UTC m=+2147.810379287" watchObservedRunningTime="2026-09-18 21:42:01.204264285 +0000 UTC m=+2158.811323842"
Sep 18 21:42:01.797691 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:42:01.797660    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:42:57.962316 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:42:57.962284    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:43:59.936565 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:43:59.936529    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:44:00.038989 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:44:00.038939    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:44:53.801887 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:44:53.801851    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:45:07.180126 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:07.180045    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:45:10.013577 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:10.013541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:45:27.862481 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:27.862447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:45:29.983823 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:29.983786    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:45:46.104550 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.104517    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"]
Sep 18 21:45:46.108065 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.108045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.129419 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.129393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/157ca5aa-95cb-4948-bcc7-f1df92f63460-openshift-service-ca\") pod \"maas-api-68c86dfd56-ldthn\" (UID: \"157ca5aa-95cb-4948-bcc7-f1df92f63460\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.129573 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.129423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/157ca5aa-95cb-4948-bcc7-f1df92f63460-maas-api-tls\") pod \"maas-api-68c86dfd56-ldthn\" (UID: \"157ca5aa-95cb-4948-bcc7-f1df92f63460\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.129573 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.129443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8hb9\" (UniqueName: \"kubernetes.io/projected/157ca5aa-95cb-4948-bcc7-f1df92f63460-kube-api-access-b8hb9\") pod \"maas-api-68c86dfd56-ldthn\" (UID: \"157ca5aa-95cb-4948-bcc7-f1df92f63460\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.129573 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.129470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/157ca5aa-95cb-4948-bcc7-f1df92f63460-metrics-certs\") pod \"maas-api-68c86dfd56-ldthn\" (UID: \"157ca5aa-95cb-4948-bcc7-f1df92f63460\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.163103 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.163068    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"]
Sep 18 21:45:46.230377 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.230343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/157ca5aa-95cb-4948-bcc7-f1df92f63460-openshift-service-ca\") pod \"maas-api-68c86dfd56-ldthn\" (UID: \"157ca5aa-95cb-4948-bcc7-f1df92f63460\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.230377 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.230378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/157ca5aa-95cb-4948-bcc7-f1df92f63460-maas-api-tls\") pod \"maas-api-68c86dfd56-ldthn\" (UID: \"157ca5aa-95cb-4948-bcc7-f1df92f63460\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.230590 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.230397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8hb9\" (UniqueName: \"kubernetes.io/projected/157ca5aa-95cb-4948-bcc7-f1df92f63460-kube-api-access-b8hb9\") pod \"maas-api-68c86dfd56-ldthn\" (UID: \"157ca5aa-95cb-4948-bcc7-f1df92f63460\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.230590 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.230424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/157ca5aa-95cb-4948-bcc7-f1df92f63460-metrics-certs\") pod \"maas-api-68c86dfd56-ldthn\" (UID: \"157ca5aa-95cb-4948-bcc7-f1df92f63460\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.231002 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.230982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/157ca5aa-95cb-4948-bcc7-f1df92f63460-openshift-service-ca\") pod \"maas-api-68c86dfd56-ldthn\" (UID: \"157ca5aa-95cb-4948-bcc7-f1df92f63460\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.232790 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.232762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/157ca5aa-95cb-4948-bcc7-f1df92f63460-metrics-certs\") pod \"maas-api-68c86dfd56-ldthn\" (UID: \"157ca5aa-95cb-4948-bcc7-f1df92f63460\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.232894 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.232809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/157ca5aa-95cb-4948-bcc7-f1df92f63460-maas-api-tls\") pod \"maas-api-68c86dfd56-ldthn\" (UID: \"157ca5aa-95cb-4948-bcc7-f1df92f63460\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.242596 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.242572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8hb9\" (UniqueName: \"kubernetes.io/projected/157ca5aa-95cb-4948-bcc7-f1df92f63460-kube-api-access-b8hb9\") pod \"maas-api-68c86dfd56-ldthn\" (UID: \"157ca5aa-95cb-4948-bcc7-f1df92f63460\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.418262 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.418178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:46.706265 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:45:46.706236    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157ca5aa_95cb_4948_bcc7_f1df92f63460.slice/crio-9404bec3df21ff7fd9947c25d00114c7567dd9ca948d17add10b8032f4e91649 WatchSource:0}: Error finding container 9404bec3df21ff7fd9947c25d00114c7567dd9ca948d17add10b8032f4e91649: Status 404 returned error can't find the container with id 9404bec3df21ff7fd9947c25d00114c7567dd9ca948d17add10b8032f4e91649
Sep 18 21:45:46.712360 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:46.712333    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"]
Sep 18 21:45:47.320711 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:47.320674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn" event={"ID":"157ca5aa-95cb-4948-bcc7-f1df92f63460","Type":"ContainerStarted","Data":"c42a6941fc9fdab817e4c6b8de37137bfe3242949db4fac48d076e4d6faf54cf"}
Sep 18 21:45:47.321215 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:47.320768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn" event={"ID":"157ca5aa-95cb-4948-bcc7-f1df92f63460","Type":"ContainerStarted","Data":"9404bec3df21ff7fd9947c25d00114c7567dd9ca948d17add10b8032f4e91649"}
Sep 18 21:45:47.321215 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:47.320889    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:47.383595 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:47.383544    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn" podStartSLOduration=0.896209231 podStartE2EDuration="1.383529372s" totalImagesPullingTime="487.320141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:45:47.381874208 +0000 UTC m=+2384.988933765" watchObservedRunningTime="2026-09-18 21:45:47.383529372 +0000 UTC m=+2384.990588958"
Sep 18 21:45:53.335864 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:53.335826    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-ldthn"
Sep 18 21:45:59.178149 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:59.178118    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:45:59.318328 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:45:59.318296    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:46:03.028405 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:03.028377    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:46:03.032093 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:03.032075    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:46:03.037729 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:03.037711    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:46:03.041223 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:03.041205    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9fpv5_9dda4277-1b86-4bad-8334-d9a06d7d9b3b/ovn-acl-logging/0.log"
Sep 18 21:46:03.630845 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:03.630810    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"]
Sep 18 21:46:03.631052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:03.631018    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827" podUID="3160642c-34c1-4164-b837-f3a13ad1f3e5" containerName="maas-api" containerID="cri-o://0935946ecbd0bd9ee775563dede549e9b9066778424c6f008dfb18c67b58e938" gracePeriod=30
Sep 18 21:46:04.866338 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:04.866312    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:46:04.995322 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:04.995248    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8kz4\" (UniqueName: \"kubernetes.io/projected/3160642c-34c1-4164-b837-f3a13ad1f3e5-kube-api-access-t8kz4\") pod \"3160642c-34c1-4164-b837-f3a13ad1f3e5\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") "
Sep 18 21:46:04.995322 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:04.995288    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3160642c-34c1-4164-b837-f3a13ad1f3e5-openshift-service-ca\") pod \"3160642c-34c1-4164-b837-f3a13ad1f3e5\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") "
Sep 18 21:46:04.995322 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:04.995323    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-maas-api-tls\") pod \"3160642c-34c1-4164-b837-f3a13ad1f3e5\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") "
Sep 18 21:46:04.995529 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:04.995352    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-metrics-certs\") pod \"3160642c-34c1-4164-b837-f3a13ad1f3e5\" (UID: \"3160642c-34c1-4164-b837-f3a13ad1f3e5\") "
Sep 18 21:46:04.995708 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:04.995685    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3160642c-34c1-4164-b837-f3a13ad1f3e5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3160642c-34c1-4164-b837-f3a13ad1f3e5" (UID: "3160642c-34c1-4164-b837-f3a13ad1f3e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:46:04.997414 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:04.997389    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3160642c-34c1-4164-b837-f3a13ad1f3e5" (UID: "3160642c-34c1-4164-b837-f3a13ad1f3e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:46:04.997414 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:04.997402    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3160642c-34c1-4164-b837-f3a13ad1f3e5-kube-api-access-t8kz4" (OuterVolumeSpecName: "kube-api-access-t8kz4") pod "3160642c-34c1-4164-b837-f3a13ad1f3e5" (UID: "3160642c-34c1-4164-b837-f3a13ad1f3e5"). InnerVolumeSpecName "kube-api-access-t8kz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:46:04.997556 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:04.997490    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3160642c-34c1-4164-b837-f3a13ad1f3e5" (UID: "3160642c-34c1-4164-b837-f3a13ad1f3e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:46:05.096820 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:05.096795    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8kz4\" (UniqueName: \"kubernetes.io/projected/3160642c-34c1-4164-b837-f3a13ad1f3e5-kube-api-access-t8kz4\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:46:05.096820 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:05.096817    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3160642c-34c1-4164-b837-f3a13ad1f3e5-openshift-service-ca\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:46:05.096972 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:05.096827    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-maas-api-tls\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:46:05.096972 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:05.096836    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3160642c-34c1-4164-b837-f3a13ad1f3e5-metrics-certs\") on node \"ip-10-0-130-92.ec2.internal\" DevicePath \"\""
Sep 18 21:46:05.407010 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:05.406975    2579 generic.go:356] "Generic (PLEG): container finished" podID="3160642c-34c1-4164-b837-f3a13ad1f3e5" containerID="0935946ecbd0bd9ee775563dede549e9b9066778424c6f008dfb18c67b58e938" exitCode=0
Sep 18 21:46:05.407177 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:05.407018    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"
Sep 18 21:46:05.407177 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:05.407019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827" event={"ID":"3160642c-34c1-4164-b837-f3a13ad1f3e5","Type":"ContainerDied","Data":"0935946ecbd0bd9ee775563dede549e9b9066778424c6f008dfb18c67b58e938"}
Sep 18 21:46:05.407177 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:05.407079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827" event={"ID":"3160642c-34c1-4164-b837-f3a13ad1f3e5","Type":"ContainerDied","Data":"64f885bb3af3868f825bf71273296abdb033f01dc0f68fc3e124dd99fe1c0702"}
Sep 18 21:46:05.407177 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:05.407101    2579 scope.go:117] "RemoveContainer" containerID="0935946ecbd0bd9ee775563dede549e9b9066778424c6f008dfb18c67b58e938"
Sep 18 21:46:05.416580 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:05.416558    2579 scope.go:117] "RemoveContainer" containerID="0935946ecbd0bd9ee775563dede549e9b9066778424c6f008dfb18c67b58e938"
Sep 18 21:46:05.416881 ip-10-0-130-92 kubenswrapper[2579]: E0918 21:46:05.416860    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0935946ecbd0bd9ee775563dede549e9b9066778424c6f008dfb18c67b58e938\": container with ID starting with 0935946ecbd0bd9ee775563dede549e9b9066778424c6f008dfb18c67b58e938 not found: ID does not exist" containerID="0935946ecbd0bd9ee775563dede549e9b9066778424c6f008dfb18c67b58e938"
Sep 18 21:46:05.416942 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:05.416889    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0935946ecbd0bd9ee775563dede549e9b9066778424c6f008dfb18c67b58e938"} err="failed to get container status \"0935946ecbd0bd9ee775563dede549e9b9066778424c6f008dfb18c67b58e938\": rpc error: code = NotFound desc = could not find container \"0935946ecbd0bd9ee775563dede549e9b9066778424c6f008dfb18c67b58e938\": container with ID starting with 0935946ecbd0bd9ee775563dede549e9b9066778424c6f008dfb18c67b58e938 not found: ID does not exist"
Sep 18 21:46:05.433793 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:05.433772    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"]
Sep 18 21:46:05.438651 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:05.438623    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5598d8c9fb-lh827"]
Sep 18 21:46:06.921971 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:06.921933    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3160642c-34c1-4164-b837-f3a13ad1f3e5" path="/var/lib/kubelet/pods/3160642c-34c1-4164-b837-f3a13ad1f3e5/volumes"
Sep 18 21:46:25.486045 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:25.485951    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-799c76f65c-h2stq_9cf727f5-89d5-4fa0-b738-54147cd30f80/manager/0.log"
Sep 18 21:46:25.912595 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:25.912559    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75f94bc69c-4rb8s_cf062a5c-8ceb-4610-972e-63c5af6b96aa/copy-manifests/0.log"
Sep 18 21:46:25.934310 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:25.934271    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75f94bc69c-4rb8s_cf062a5c-8ceb-4610-972e-63c5af6b96aa/manager/0.log"
Sep 18 21:46:26.598785 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:26.598753    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68c86dfd56-ldthn_157ca5aa-95cb-4948-bcc7-f1df92f63460/maas-api/0.log"
Sep 18 21:46:27.132941 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:27.132904    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-4qccc_4bafa5e1-5077-47d1-aeff-996533129bcc/postgres/0.log"
Sep 18 21:46:28.778818 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:28.778788    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:46:29.527152 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:29.527120    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85cdd48bb5-vg2xq_90b1e7d0-2639-47b4-ba8d-815c44e99e30/kube-auth-proxy/0.log"
Sep 18 21:46:29.924076 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:29.924046    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9kv57_0392321b-d271-4c1c-beea-5c2e7d5802db/payload-processing/0.log"
Sep 18 21:46:30.453945 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:30.453917    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-gg8wp_384a741b-99f9-45c6-8596-f927061235ad/main/0.log"
Sep 18 21:46:30.979206 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:30.979172    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fv5h92_2d809256-f651-48e0-b3ba-f62340bed671/main/0.log"
Sep 18 21:46:31.233698 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:31.233600    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-hfpcn_63cdd729-6d75-4d94-bab1-0c1c5c7068be/main/0.log"
Sep 18 21:46:36.451876 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:36.451840    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:46:36.587151 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:36.587122    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:46:39.861919 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:39.861890    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rzd48_5b2108fb-5c03-49d2-a864-2ef38a1fbaab/global-pull-secret-syncer/0.log"
Sep 18 21:46:39.934664 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:39.934622    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log"
Sep 18 21:46:40.118968 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:40.118889    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-92.ec2.internal_c6ac307dfcd56096fce875fa1e1ed1ca/haproxy/0.log"
Sep 18 21:46:46.635819 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:46.635787    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae6fa83-229b-4b0c-b6fd-4d39f54041fd/alertmanager/0.log"
Sep 18 21:46:46.723052 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:46.723020    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae6fa83-229b-4b0c-b6fd-4d39f54041fd/config-reloader/0.log"
Sep 18 21:46:46.834297 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:46.834270    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae6fa83-229b-4b0c-b6fd-4d39f54041fd/kube-rbac-proxy-web/0.log"
Sep 18 21:46:46.946551 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:46.946470    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae6fa83-229b-4b0c-b6fd-4d39f54041fd/kube-rbac-proxy/0.log"
Sep 18 21:46:47.077202 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:47.077164    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae6fa83-229b-4b0c-b6fd-4d39f54041fd/kube-rbac-proxy-metric/0.log"
Sep 18 21:46:47.185199 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:47.185166    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae6fa83-229b-4b0c-b6fd-4d39f54041fd/prom-label-proxy/0.log"
Sep 18 21:46:47.371494 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:47.371464    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ae6fa83-229b-4b0c-b6fd-4d39f54041fd/init-config-reloader/0.log"
Sep 18 21:46:47.631870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:47.631843    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-lh95h_153a9d01-c1ff-471d-a43e-9387dc4917e5/kube-state-metrics/0.log"
Sep 18 21:46:47.774728 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:47.774694    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-lh95h_153a9d01-c1ff-471d-a43e-9387dc4917e5/kube-rbac-proxy-main/0.log"
Sep 18 21:46:47.985739 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:47.985670    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-lh95h_153a9d01-c1ff-471d-a43e-9387dc4917e5/kube-rbac-proxy-self/0.log"
Sep 18 21:46:48.078076 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:48.078050    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6467685b8-xgnwr_329862e1-1940-4da0-9cca-483aec6e3486/metrics-server/0.log"
Sep 18 21:46:48.730532 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:48.730457    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-scjlc_d27c900a-494e-4028-b6bc-ed4794f33e63/node-exporter/0.log"
Sep 18 21:46:48.815963 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:48.815939    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-scjlc_d27c900a-494e-4028-b6bc-ed4794f33e63/kube-rbac-proxy/0.log"
Sep 18 21:46:48.879194 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:48.879172    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-scjlc_d27c900a-494e-4028-b6bc-ed4794f33e63/init-textfile/0.log"
Sep 18 21:46:49.009413 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.009332    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"]
Sep 18 21:46:49.009735 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.009722    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3160642c-34c1-4164-b837-f3a13ad1f3e5" containerName="maas-api"
Sep 18 21:46:49.009789 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.009737    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3160642c-34c1-4164-b837-f3a13ad1f3e5" containerName="maas-api"
Sep 18 21:46:49.009825 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.009803    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3160642c-34c1-4164-b837-f3a13ad1f3e5" containerName="maas-api"
Sep 18 21:46:49.012924 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.012906    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.015897 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.015873    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s7tjl\"/\"openshift-service-ca.crt\""
Sep 18 21:46:49.016000 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.015912    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s7tjl\"/\"kube-root-ca.crt\""
Sep 18 21:46:49.017090 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.017071    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s7tjl\"/\"default-dockercfg-jr7nq\""
Sep 18 21:46:49.041101 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.041071    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"]
Sep 18 21:46:49.100181 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.100146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f53314a-6887-4539-813c-ed39005f9bc5-podres\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.100362 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.100190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f53314a-6887-4539-813c-ed39005f9bc5-sys\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.100362 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.100232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f53314a-6887-4539-813c-ed39005f9bc5-lib-modules\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.100362 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.100265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f53314a-6887-4539-813c-ed39005f9bc5-proc\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.100491 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.100366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c67zb\" (UniqueName: \"kubernetes.io/projected/1f53314a-6887-4539-813c-ed39005f9bc5-kube-api-access-c67zb\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.168135 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.168099    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_784f9aa2-4755-4362-b9fb-76b1f190124d/prometheus/0.log"
Sep 18 21:46:49.201769 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.201737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f53314a-6887-4539-813c-ed39005f9bc5-podres\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.201950 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.201782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f53314a-6887-4539-813c-ed39005f9bc5-sys\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.201950 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.201826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f53314a-6887-4539-813c-ed39005f9bc5-lib-modules\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.201950 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.201863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f53314a-6887-4539-813c-ed39005f9bc5-proc\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.201950 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.201907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f53314a-6887-4539-813c-ed39005f9bc5-podres\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.201950 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.201924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f53314a-6887-4539-813c-ed39005f9bc5-sys\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.201950 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.201927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c67zb\" (UniqueName: \"kubernetes.io/projected/1f53314a-6887-4539-813c-ed39005f9bc5-kube-api-access-c67zb\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.202249 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.201965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f53314a-6887-4539-813c-ed39005f9bc5-lib-modules\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.202249 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.201982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f53314a-6887-4539-813c-ed39005f9bc5-proc\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.213245 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.213213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c67zb\" (UniqueName: \"kubernetes.io/projected/1f53314a-6887-4539-813c-ed39005f9bc5-kube-api-access-c67zb\") pod \"perf-node-gather-daemonset-m4v79\" (UID: \"1f53314a-6887-4539-813c-ed39005f9bc5\") " pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.225133 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.225104    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_784f9aa2-4755-4362-b9fb-76b1f190124d/config-reloader/0.log"
Sep 18 21:46:49.323870 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.323832    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:49.476531 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.476494    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_784f9aa2-4755-4362-b9fb-76b1f190124d/thanos-sidecar/0.log"
Sep 18 21:46:49.637361 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.637331    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"]
Sep 18 21:46:49.640280 ip-10-0-130-92 kubenswrapper[2579]: W0918 21:46:49.640257    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1f53314a_6887_4539_813c_ed39005f9bc5.slice/crio-beb4c11f58d88f8c4af0e9d1bbac7b862e0cde38e567da6460c742a02882d92a WatchSource:0}: Error finding container beb4c11f58d88f8c4af0e9d1bbac7b862e0cde38e567da6460c742a02882d92a: Status 404 returned error can't find the container with id beb4c11f58d88f8c4af0e9d1bbac7b862e0cde38e567da6460c742a02882d92a
Sep 18 21:46:49.697762 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.697740    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_784f9aa2-4755-4362-b9fb-76b1f190124d/kube-rbac-proxy-web/0.log"
Sep 18 21:46:49.786490 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.786459    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_784f9aa2-4755-4362-b9fb-76b1f190124d/kube-rbac-proxy/0.log"
Sep 18 21:46:49.832173 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.832147    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_784f9aa2-4755-4362-b9fb-76b1f190124d/kube-rbac-proxy-thanos/0.log"
Sep 18 21:46:49.890129 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:49.890057    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_784f9aa2-4755-4362-b9fb-76b1f190124d/init-config-reloader/0.log"
Sep 18 21:46:50.156204 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:50.156123    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-mqvzc_bfee6987-d86e-440b-b097-0e4b5539d8c4/prometheus-operator-admission-webhook/0.log"
Sep 18 21:46:50.236351 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:50.236309    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58d88548b5-f4m8f_dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76/telemeter-client/0.log"
Sep 18 21:46:50.369702 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:50.369665    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58d88548b5-f4m8f_dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76/reload/0.log"
Sep 18 21:46:50.421549 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:50.421478    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58d88548b5-f4m8f_dc8fcf08-aaa8-4dcc-8fed-3ec01be70c76/kube-rbac-proxy/0.log"
Sep 18 21:46:50.628049 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:50.628016    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79" event={"ID":"1f53314a-6887-4539-813c-ed39005f9bc5","Type":"ContainerStarted","Data":"ade9cd1812a8a4bfcfc59ff1abb27a460503e99d1d107c2928e61d313d7e5276"}
Sep 18 21:46:50.628207 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:50.628066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79" event={"ID":"1f53314a-6887-4539-813c-ed39005f9bc5","Type":"ContainerStarted","Data":"beb4c11f58d88f8c4af0e9d1bbac7b862e0cde38e567da6460c742a02882d92a"}
Sep 18 21:46:50.628207 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:50.628163    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:50.676948 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:50.676838    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79" podStartSLOduration=2.676821145 podStartE2EDuration="2.676821145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:46:50.675614912 +0000 UTC m=+2448.282674470" watchObservedRunningTime="2026-09-18 21:46:50.676821145 +0000 UTC m=+2448.283880722"
Sep 18 21:46:52.767456 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:52.767399    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/1.log"
Sep 18 21:46:52.772249 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:52.772202    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jgkcm_1fbb0af0-328b-447e-824c-0d9d74840047/console-operator/2.log"
Sep 18 21:46:53.520951 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:53.520916    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-pdrnp_22b37c6f-d508-4c3e-a969-238c8900482c/download-server/0.log"
Sep 18 21:46:56.037827 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:56.037798    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9rc6_d1225d31-f513-4cb7-b21a-a97accdb1bfb/dns/0.log"
Sep 18 21:46:56.192348 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:56.192319    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9rc6_d1225d31-f513-4cb7-b21a-a97accdb1bfb/kube-rbac-proxy/0.log"
Sep 18 21:46:56.386075 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:56.385998    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-khlzv_13cbaf64-d066-42f3-a262-8e56d161434a/dns-node-resolver/0.log"
Sep 18 21:46:56.642927 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:56.642842    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s7tjl/perf-node-gather-daemonset-m4v79"
Sep 18 21:46:57.003073 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:57.003003    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-27drt_63bb4238-7a5e-462c-9b50-45e3b2813957/konnectivity-agent/0.log" limit=1024
Sep 18 21:46:57.286612 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:57.286588    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tkw2z_dc1f208a-bfe4-49d1-9dac-1214ace566d8/node-ca/0.log"
Sep 18 21:46:59.902246 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:46:59.902218    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85cdd48bb5-vg2xq_90b1e7d0-2639-47b4-ba8d-815c44e99e30/kube-auth-proxy/0.log"
Sep 18 21:47:00.097078 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:47:00.097049    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9kv57_0392321b-d271-4c1c-beea-5c2e7d5802db/payload-processing/0.log"
Sep 18 21:47:01.261136 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:47:01.261110    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85cdd48bb5-vg2xq_90b1e7d0-2639-47b4-ba8d-815c44e99e30/kube-auth-proxy/0.log"
Sep 18 21:47:01.473038 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:47:01.472983    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9kv57_0392321b-d271-4c1c-beea-5c2e7d5802db/payload-processing/0.log"
Sep 18 21:47:01.483341 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:47:01.483318    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4brns_203754e1-73e8-415d-be7a-f187ea164375/serve-healthcheck-canary/0.log"
Sep 18 21:47:02.530660 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:47:02.530611    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j4x8n_82e6e1c8-fb55-4eff-b955-6376f98c29bc/kube-rbac-proxy/0.log"
Sep 18 21:47:02.573721 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:47:02.573698    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j4x8n_82e6e1c8-fb55-4eff-b955-6376f98c29bc/exporter/0.log"
Sep 18 21:47:02.637226 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:47:02.637199    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j4x8n_82e6e1c8-fb55-4eff-b955-6376f98c29bc/extractor/0.log"
Sep 18 21:47:03.695717 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:47:03.695686    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85cdd48bb5-vg2xq_90b1e7d0-2639-47b4-ba8d-815c44e99e30/kube-auth-proxy/0.log"
Sep 18 21:47:03.880710 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:47:03.880669    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9kv57_0392321b-d271-4c1c-beea-5c2e7d5802db/payload-processing/0.log"
Sep 18 21:47:06.186027 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:47:06.185925    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-799c76f65c-h2stq_9cf727f5-89d5-4fa0-b738-54147cd30f80/manager/0.log"
Sep 18 21:47:06.569348 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:47:06.569315    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75f94bc69c-4rb8s_cf062a5c-8ceb-4610-972e-63c5af6b96aa/manager/0.log"
Sep 18 21:47:06.695273 ip-10-0-130-92 kubenswrapper[2579]: I0918 21:47:06.695246    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75f94bc69c-4rb8s_cf062a5c-8ceb-4610-972e-63c5af6b96aa/copy-manifests/0.log"